{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using LightGBM classifier on credit card user data to predict default rate\n",
    "\n",
    "This notebook utilizes the LightGBM package by Microsoft with a clean dataset on credit card defaults (source: https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients). Since we are trying to predict if a person is defaulting or not, we will want to use binary classification. This notebook also explores early stopping, a form of regularization used to avoid overfitting when training a learner with an iterative method, such as gradient descent. Such methods update the learner so as to make it better fit the training data with each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import lightgbm as lgb\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import auc, accuracy_score, roc_auc_score, roc_curve\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_excel('ccdata.xls', header = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>LIMIT_BAL</th>\n",
       "      <th>SEX</th>\n",
       "      <th>EDUCATION</th>\n",
       "      <th>MARRIAGE</th>\n",
       "      <th>AGE</th>\n",
       "      <th>PAY_0</th>\n",
       "      <th>PAY_2</th>\n",
       "      <th>PAY_3</th>\n",
       "      <th>PAY_4</th>\n",
       "      <th>...</th>\n",
       "      <th>BILL_AMT4</th>\n",
       "      <th>BILL_AMT5</th>\n",
       "      <th>BILL_AMT6</th>\n",
       "      <th>PAY_AMT1</th>\n",
       "      <th>PAY_AMT2</th>\n",
       "      <th>PAY_AMT3</th>\n",
       "      <th>PAY_AMT4</th>\n",
       "      <th>PAY_AMT5</th>\n",
       "      <th>PAY_AMT6</th>\n",
       "      <th>default payment next month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>20000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>689</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>120000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>26</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>3272</td>\n",
       "      <td>3455</td>\n",
       "      <td>3261</td>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>0</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>90000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>14331</td>\n",
       "      <td>14948</td>\n",
       "      <td>15549</td>\n",
       "      <td>1518</td>\n",
       "      <td>1500</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>50000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>28314</td>\n",
       "      <td>28959</td>\n",
       "      <td>29547</td>\n",
       "      <td>2000</td>\n",
       "      <td>2019</td>\n",
       "      <td>1200</td>\n",
       "      <td>1100</td>\n",
       "      <td>1069</td>\n",
       "      <td>1000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>50000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>57</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>20940</td>\n",
       "      <td>19146</td>\n",
       "      <td>19131</td>\n",
       "      <td>2000</td>\n",
       "      <td>36681</td>\n",
       "      <td>10000</td>\n",
       "      <td>9000</td>\n",
       "      <td>689</td>\n",
       "      <td>679</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   ID  LIMIT_BAL  SEX  EDUCATION  MARRIAGE  AGE  PAY_0  PAY_2  PAY_3  PAY_4  \\\n",
       "0   1      20000    2          2         1   24      2      2     -1     -1   \n",
       "1   2     120000    2          2         2   26     -1      2      0      0   \n",
       "2   3      90000    2          2         2   34      0      0      0      0   \n",
       "3   4      50000    2          2         1   37      0      0      0      0   \n",
       "4   5      50000    1          2         1   57     -1      0     -1      0   \n",
       "\n",
       "   ...  BILL_AMT4  BILL_AMT5  BILL_AMT6  PAY_AMT1  PAY_AMT2  PAY_AMT3  \\\n",
       "0  ...          0          0          0         0       689         0   \n",
       "1  ...       3272       3455       3261         0      1000      1000   \n",
       "2  ...      14331      14948      15549      1518      1500      1000   \n",
       "3  ...      28314      28959      29547      2000      2019      1200   \n",
       "4  ...      20940      19146      19131      2000     36681     10000   \n",
       "\n",
       "   PAY_AMT4  PAY_AMT5  PAY_AMT6  default payment next month  \n",
       "0         0         0         0                           1  \n",
       "1      1000         0      2000                           1  \n",
       "2      1000      1000      5000                           0  \n",
       "3      1100      1069      1000                           0  \n",
       "4      9000       689       679                           0  \n",
       "\n",
       "[5 rows x 25 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This research employed a binary variable, default payment (Yes = 1, No = 0), as the response variable. This study reviewed the literature and used the following 23 variables as explanatory variables:\n",
    "- X1: Amount of the given credit (NT dollar): it includes both the individual consumer credit and his/her family (supplementary) credit.\n",
    "- X2: Gender (1 = male; 2 = female).\n",
    "- X3: Education (1 = graduate school; 2 = university; 3 = high school; 4 = others).\n",
    "- X4: Marital status (1 = married; 2 = single; 3 = others).\n",
    "- X5: Age (year).\n",
    "- X6 - X11: History of past payment. We tracked the past monthly payment records (from April to September, 2005) as follows: X6 = the repayment status in September, 2005; X7 = the repayment status in August, 2005; . . .;X11 = the repayment status in April, 2005. The measurement scale for the repayment status is: -1 = pay duly; 1 = payment delay for one month; 2 = payment delay for two months; . . .; 8 = payment delay for eight months; 9 = payment delay for nine months and above.\n",
    "- X12-X17: Amount of bill statement (NT dollar). X12 = amount of bill statement in September, 2005; X13 = amount of bill statement in August, 2005; . . .; X17 = amount of bill statement in April, 2005.\n",
    "- X18-X23: Amount of previous payment (NT dollar). X18 = amount paid in September, 2005; X19 = amount paid in August, 2005; . . .;X23 = amount paid in April, 2005.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.drop('ID', axis = 1, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "default payment next month    0\n",
       "PAY_AMT6                      0\n",
       "SEX                           0\n",
       "EDUCATION                     0\n",
       "MARRIAGE                      0\n",
       "AGE                           0\n",
       "PAY_0                         0\n",
       "PAY_2                         0\n",
       "PAY_3                         0\n",
       "PAY_4                         0\n",
       "PAY_5                         0\n",
       "PAY_6                         0\n",
       "BILL_AMT1                     0\n",
       "BILL_AMT2                     0\n",
       "BILL_AMT3                     0\n",
       "BILL_AMT4                     0\n",
       "BILL_AMT5                     0\n",
       "BILL_AMT6                     0\n",
       "PAY_AMT1                      0\n",
       "PAY_AMT2                      0\n",
       "PAY_AMT3                      0\n",
       "PAY_AMT4                      0\n",
       "PAY_AMT5                      0\n",
       "LIMIT_BAL                     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for null values. \n",
    "data.isnull().sum().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thank god there are no missing values! If there were, we would have to either drop the feature if \n",
    "there are more than 60% of the values are missing, or impute them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Outlier Detection in this cell.\n",
    "# For each column, first it computes the Z-score of each value in the column,\n",
    "# relative to the column mean and standard deviation. Then is takes the absolute \n",
    "# of Z-score because the direction does not matter, only if it is below the threshold.\n",
    "# all(axis=1) ensures that for each row, all column satisfy the constraint. \n",
    "data = data[data.apply(lambda x: np.abs(x - x.mean()) / x.std() < 3).all(axis=1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Below, we separate our target variable from our dataset. \n",
    "X = data.drop(['default payment next month'], axis=1)\n",
    "y = data['default payment next month']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22464, 23)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "\n",
    "\n",
    "# Let's then put 85% of our dataset into a training set and 15% of it into a test set. \n",
    "# We can use random_state because it is not a time series dataset that we are using.\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=20)\n",
    "X_train.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>LightGBM classifier hyperparameter optimization via scikit-learn's GridSearchCV</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ajc/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_split.py:2053: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.650853\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.616135\tvalid_0's auc: 0.752078\n",
      "[3]\tvalid_0's binary_logloss: 0.587752\tvalid_0's auc: 0.755301\n",
      "[4]\tvalid_0's binary_logloss: 0.564477\tvalid_0's auc: 0.756622\n",
      "[5]\tvalid_0's binary_logloss: 0.544594\tvalid_0's auc: 0.760193\n",
      "[6]\tvalid_0's binary_logloss: 0.527785\tvalid_0's auc: 0.760906\n",
      "[7]\tvalid_0's binary_logloss: 0.513856\tvalid_0's auc: 0.761905\n",
      "[8]\tvalid_0's binary_logloss: 0.502076\tvalid_0's auc: 0.76343\n",
      "[9]\tvalid_0's binary_logloss: 0.491974\tvalid_0's auc: 0.763981\n",
      "[10]\tvalid_0's binary_logloss: 0.483799\tvalid_0's auc: 0.763893\n",
      "[11]\tvalid_0's binary_logloss: 0.476724\tvalid_0's auc: 0.765105\n",
      "[12]\tvalid_0's binary_logloss: 0.470592\tvalid_0's auc: 0.766134\n",
      "[13]\tvalid_0's binary_logloss: 0.465488\tvalid_0's auc: 0.766579\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[15]\tvalid_0's binary_logloss: 0.458058\tvalid_0's auc: 0.765453\n",
      "[16]\tvalid_0's binary_logloss: 0.455224\tvalid_0's auc: 0.765914\n",
      "[17]\tvalid_0's binary_logloss: 0.452883\tvalid_0's auc: 0.765398\n",
      "[18]\tvalid_0's binary_logloss: 0.450593\tvalid_0's auc: 0.766378\n",
      "[19]\tvalid_0's binary_logloss: 0.44907\tvalid_0's auc: 0.765987\n",
      "Early stopping, best iteration is:\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[1]\tvalid_0's binary_logloss: 0.650327\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.61564\tvalid_0's auc: 0.757175\n",
      "[3]\tvalid_0's binary_logloss: 0.586972\tvalid_0's auc: 0.764043\n",
      "[4]\tvalid_0's binary_logloss: 0.562638\tvalid_0's auc: 0.766654\n",
      "[5]\tvalid_0's binary_logloss: 0.542872\tvalid_0's auc: 0.766568\n",
      "[6]\tvalid_0's binary_logloss: 0.52609\tvalid_0's auc: 0.768887\n",
      "[7]\tvalid_0's binary_logloss: 0.512487\tvalid_0's auc: 0.769703\n",
      "[8]\tvalid_0's binary_logloss: 0.500321\tvalid_0's auc: 0.771338\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[10]\tvalid_0's binary_logloss: 0.481959\tvalid_0's auc: 0.770874\n",
      "[11]\tvalid_0's binary_logloss: 0.475167\tvalid_0's auc: 0.771372\n",
      "[12]\tvalid_0's binary_logloss: 0.46925\tvalid_0's auc: 0.770875\n",
      "[13]\tvalid_0's binary_logloss: 0.464184\tvalid_0's auc: 0.771062\n",
      "[14]\tvalid_0's binary_logloss: 0.460043\tvalid_0's auc: 0.770341\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[1]\tvalid_0's binary_logloss: 0.650309\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.615779\tvalid_0's auc: 0.756524\n",
      "[3]\tvalid_0's binary_logloss: 0.586994\tvalid_0's auc: 0.759714\n",
      "[4]\tvalid_0's binary_logloss: 0.563245\tvalid_0's auc: 0.761988\n",
      "[5]\tvalid_0's binary_logloss: 0.54326\tvalid_0's auc: 0.763567\n",
      "[6]\tvalid_0's binary_logloss: 0.526682\tvalid_0's auc: 0.762949\n",
      "[7]\tvalid_0's binary_logloss: 0.51248\tvalid_0's auc: 0.764754\n",
      "[8]\tvalid_0's binary_logloss: 0.500651\tvalid_0's auc: 0.764875\n",
      "[9]\tvalid_0's binary_logloss: 0.49096\tvalid_0's auc: 0.76325\n",
      "[10]\tvalid_0's binary_logloss: 0.482827\tvalid_0's auc: 0.763713\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[12]\tvalid_0's binary_logloss: 0.470064\tvalid_0's auc: 0.76424\n",
      "[13]\tvalid_0's binary_logloss: 0.464881\tvalid_0's auc: 0.765382\n",
      "[14]\tvalid_0's binary_logloss: 0.460645\tvalid_0's auc: 0.765338\n",
      "[15]\tvalid_0's binary_logloss: 0.457268\tvalid_0's auc: 0.765179\n",
      "[16]\tvalid_0's binary_logloss: 0.454458\tvalid_0's auc: 0.76512\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[1]\tvalid_0's binary_logloss: 0.650853\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.616135\tvalid_0's auc: 0.752078\n",
      "[3]\tvalid_0's binary_logloss: 0.587752\tvalid_0's auc: 0.755301\n",
      "[4]\tvalid_0's binary_logloss: 0.564477\tvalid_0's auc: 0.756622\n",
      "[5]\tvalid_0's binary_logloss: 0.544594\tvalid_0's auc: 0.760193\n",
      "[6]\tvalid_0's binary_logloss: 0.527785\tvalid_0's auc: 0.760906\n",
      "[7]\tvalid_0's binary_logloss: 0.513856\tvalid_0's auc: 0.761905\n",
      "[8]\tvalid_0's binary_logloss: 0.502076\tvalid_0's auc: 0.76343\n",
      "[9]\tvalid_0's binary_logloss: 0.491974\tvalid_0's auc: 0.763981\n",
      "[10]\tvalid_0's binary_logloss: 0.483799\tvalid_0's auc: 0.763893\n",
      "[11]\tvalid_0's binary_logloss: 0.476724\tvalid_0's auc: 0.765105\n",
      "[12]\tvalid_0's binary_logloss: 0.470592\tvalid_0's auc: 0.766134\n",
      "[13]\tvalid_0's binary_logloss: 0.465488\tvalid_0's auc: 0.766579\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[15]\tvalid_0's binary_logloss: 0.458058\tvalid_0's auc: 0.765453\n",
      "[16]\tvalid_0's binary_logloss: 0.455224\tvalid_0's auc: 0.765914\n",
      "[17]\tvalid_0's binary_logloss: 0.452883\tvalid_0's auc: 0.765398\n",
      "[18]\tvalid_0's binary_logloss: 0.450593\tvalid_0's auc: 0.766378\n",
      "[19]\tvalid_0's binary_logloss: 0.44907\tvalid_0's auc: 0.765987\n",
      "Early stopping, best iteration is:\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[1]\tvalid_0's binary_logloss: 0.650327\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.61564\tvalid_0's auc: 0.757175\n",
      "[3]\tvalid_0's binary_logloss: 0.586972\tvalid_0's auc: 0.764043\n",
      "[4]\tvalid_0's binary_logloss: 0.562638\tvalid_0's auc: 0.766654\n",
      "[5]\tvalid_0's binary_logloss: 0.542872\tvalid_0's auc: 0.766568\n",
      "[6]\tvalid_0's binary_logloss: 0.52609\tvalid_0's auc: 0.768887\n",
      "[7]\tvalid_0's binary_logloss: 0.512487\tvalid_0's auc: 0.769703\n",
      "[8]\tvalid_0's binary_logloss: 0.500321\tvalid_0's auc: 0.771338\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[10]\tvalid_0's binary_logloss: 0.481959\tvalid_0's auc: 0.770874\n",
      "[11]\tvalid_0's binary_logloss: 0.475167\tvalid_0's auc: 0.771372\n",
      "[12]\tvalid_0's binary_logloss: 0.46925\tvalid_0's auc: 0.770875\n",
      "[13]\tvalid_0's binary_logloss: 0.464184\tvalid_0's auc: 0.771062\n",
      "[14]\tvalid_0's binary_logloss: 0.460043\tvalid_0's auc: 0.770341\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[1]\tvalid_0's binary_logloss: 0.650309\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.615779\tvalid_0's auc: 0.756524\n",
      "[3]\tvalid_0's binary_logloss: 0.586994\tvalid_0's auc: 0.759714\n",
      "[4]\tvalid_0's binary_logloss: 0.563245\tvalid_0's auc: 0.761988\n",
      "[5]\tvalid_0's binary_logloss: 0.54326\tvalid_0's auc: 0.763567\n",
      "[6]\tvalid_0's binary_logloss: 0.526682\tvalid_0's auc: 0.762949\n",
      "[7]\tvalid_0's binary_logloss: 0.51248\tvalid_0's auc: 0.764754\n",
      "[8]\tvalid_0's binary_logloss: 0.500651\tvalid_0's auc: 0.764875\n",
      "[9]\tvalid_0's binary_logloss: 0.49096\tvalid_0's auc: 0.76325\n",
      "[10]\tvalid_0's binary_logloss: 0.482827\tvalid_0's auc: 0.763713\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[12]\tvalid_0's binary_logloss: 0.470064\tvalid_0's auc: 0.76424\n",
      "[13]\tvalid_0's binary_logloss: 0.464881\tvalid_0's auc: 0.765382\n",
      "[14]\tvalid_0's binary_logloss: 0.460645\tvalid_0's auc: 0.765338\n",
      "[15]\tvalid_0's binary_logloss: 0.457268\tvalid_0's auc: 0.765179\n",
      "[16]\tvalid_0's binary_logloss: 0.454458\tvalid_0's auc: 0.76512\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[1]\tvalid_0's binary_logloss: 0.650853\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.616135\tvalid_0's auc: 0.752078\n",
      "[3]\tvalid_0's binary_logloss: 0.587752\tvalid_0's auc: 0.755301\n",
      "[4]\tvalid_0's binary_logloss: 0.564477\tvalid_0's auc: 0.756622\n",
      "[5]\tvalid_0's binary_logloss: 0.544594\tvalid_0's auc: 0.760193\n",
      "[6]\tvalid_0's binary_logloss: 0.527785\tvalid_0's auc: 0.760906\n",
      "[7]\tvalid_0's binary_logloss: 0.513856\tvalid_0's auc: 0.761905\n",
      "[8]\tvalid_0's binary_logloss: 0.502076\tvalid_0's auc: 0.76343\n",
      "[9]\tvalid_0's binary_logloss: 0.491974\tvalid_0's auc: 0.763981\n",
      "[10]\tvalid_0's binary_logloss: 0.483799\tvalid_0's auc: 0.763893\n",
      "[11]\tvalid_0's binary_logloss: 0.476724\tvalid_0's auc: 0.765105\n",
      "[12]\tvalid_0's binary_logloss: 0.470592\tvalid_0's auc: 0.766134\n",
      "[13]\tvalid_0's binary_logloss: 0.465488\tvalid_0's auc: 0.766579\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[15]\tvalid_0's binary_logloss: 0.458058\tvalid_0's auc: 0.765453\n",
      "[16]\tvalid_0's binary_logloss: 0.455224\tvalid_0's auc: 0.765914\n",
      "[17]\tvalid_0's binary_logloss: 0.452883\tvalid_0's auc: 0.765398\n",
      "[18]\tvalid_0's binary_logloss: 0.450593\tvalid_0's auc: 0.766378\n",
      "[19]\tvalid_0's binary_logloss: 0.44907\tvalid_0's auc: 0.765987\n",
      "Early stopping, best iteration is:\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.650327\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.61564\tvalid_0's auc: 0.757175\n",
      "[3]\tvalid_0's binary_logloss: 0.586972\tvalid_0's auc: 0.764043\n",
      "[4]\tvalid_0's binary_logloss: 0.562638\tvalid_0's auc: 0.766654\n",
      "[5]\tvalid_0's binary_logloss: 0.542872\tvalid_0's auc: 0.766568\n",
      "[6]\tvalid_0's binary_logloss: 0.52609\tvalid_0's auc: 0.768887\n",
      "[7]\tvalid_0's binary_logloss: 0.512487\tvalid_0's auc: 0.769703\n",
      "[8]\tvalid_0's binary_logloss: 0.500321\tvalid_0's auc: 0.771338\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[10]\tvalid_0's binary_logloss: 0.481959\tvalid_0's auc: 0.770874\n",
      "[11]\tvalid_0's binary_logloss: 0.475167\tvalid_0's auc: 0.771372\n",
      "[12]\tvalid_0's binary_logloss: 0.46925\tvalid_0's auc: 0.770875\n",
      "[13]\tvalid_0's binary_logloss: 0.464184\tvalid_0's auc: 0.771062\n",
      "[14]\tvalid_0's binary_logloss: 0.460043\tvalid_0's auc: 0.770341\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[1]\tvalid_0's binary_logloss: 0.650309\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.615779\tvalid_0's auc: 0.756524\n",
      "[3]\tvalid_0's binary_logloss: 0.586994\tvalid_0's auc: 0.759714\n",
      "[4]\tvalid_0's binary_logloss: 0.563245\tvalid_0's auc: 0.761988\n",
      "[5]\tvalid_0's binary_logloss: 0.54326\tvalid_0's auc: 0.763567\n",
      "[6]\tvalid_0's binary_logloss: 0.526682\tvalid_0's auc: 0.762949\n",
      "[7]\tvalid_0's binary_logloss: 0.51248\tvalid_0's auc: 0.764754\n",
      "[8]\tvalid_0's binary_logloss: 0.500651\tvalid_0's auc: 0.764875\n",
      "[9]\tvalid_0's binary_logloss: 0.49096\tvalid_0's auc: 0.76325\n",
      "[10]\tvalid_0's binary_logloss: 0.482827\tvalid_0's auc: 0.763713\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[12]\tvalid_0's binary_logloss: 0.470064\tvalid_0's auc: 0.76424\n",
      "[13]\tvalid_0's binary_logloss: 0.464881\tvalid_0's auc: 0.765382\n",
      "[14]\tvalid_0's binary_logloss: 0.460645\tvalid_0's auc: 0.765338\n",
      "[15]\tvalid_0's binary_logloss: 0.457268\tvalid_0's auc: 0.765179\n",
      "[16]\tvalid_0's binary_logloss: 0.454458\tvalid_0's auc: 0.76512\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[1]\tvalid_0's binary_logloss: 0.650853\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.616135\tvalid_0's auc: 0.752078\n",
      "[3]\tvalid_0's binary_logloss: 0.587752\tvalid_0's auc: 0.755301\n",
      "[4]\tvalid_0's binary_logloss: 0.564477\tvalid_0's auc: 0.756622\n",
      "[5]\tvalid_0's binary_logloss: 0.544594\tvalid_0's auc: 0.760193\n",
      "[6]\tvalid_0's binary_logloss: 0.527785\tvalid_0's auc: 0.760906\n",
      "[7]\tvalid_0's binary_logloss: 0.513856\tvalid_0's auc: 0.761905\n",
      "[8]\tvalid_0's binary_logloss: 0.502076\tvalid_0's auc: 0.76343\n",
      "[9]\tvalid_0's binary_logloss: 0.491974\tvalid_0's auc: 0.763981\n",
      "[10]\tvalid_0's binary_logloss: 0.483799\tvalid_0's auc: 0.763893\n",
      "[11]\tvalid_0's binary_logloss: 0.476724\tvalid_0's auc: 0.765105\n",
      "[12]\tvalid_0's binary_logloss: 0.470592\tvalid_0's auc: 0.766134\n",
      "[13]\tvalid_0's binary_logloss: 0.465488\tvalid_0's auc: 0.766579\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[15]\tvalid_0's binary_logloss: 0.458058\tvalid_0's auc: 0.765453\n",
      "[16]\tvalid_0's binary_logloss: 0.455224\tvalid_0's auc: 0.765914\n",
      "[17]\tvalid_0's binary_logloss: 0.452883\tvalid_0's auc: 0.765398\n",
      "[18]\tvalid_0's binary_logloss: 0.450593\tvalid_0's auc: 0.766378\n",
      "[19]\tvalid_0's binary_logloss: 0.44907\tvalid_0's auc: 0.765987\n",
      "Early stopping, best iteration is:\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[1]\tvalid_0's binary_logloss: 0.650327\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.61564\tvalid_0's auc: 0.757175\n",
      "[3]\tvalid_0's binary_logloss: 0.586972\tvalid_0's auc: 0.764043\n",
      "[4]\tvalid_0's binary_logloss: 0.562638\tvalid_0's auc: 0.766654\n",
      "[5]\tvalid_0's binary_logloss: 0.542872\tvalid_0's auc: 0.766568\n",
      "[6]\tvalid_0's binary_logloss: 0.52609\tvalid_0's auc: 0.768887\n",
      "[7]\tvalid_0's binary_logloss: 0.512487\tvalid_0's auc: 0.769703\n",
      "[8]\tvalid_0's binary_logloss: 0.500321\tvalid_0's auc: 0.771338\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[10]\tvalid_0's binary_logloss: 0.481959\tvalid_0's auc: 0.770874\n",
      "[11]\tvalid_0's binary_logloss: 0.475167\tvalid_0's auc: 0.771372\n",
      "[12]\tvalid_0's binary_logloss: 0.46925\tvalid_0's auc: 0.770875\n",
      "[13]\tvalid_0's binary_logloss: 0.464184\tvalid_0's auc: 0.771062\n",
      "[14]\tvalid_0's binary_logloss: 0.460043\tvalid_0's auc: 0.770341\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[1]\tvalid_0's binary_logloss: 0.650309\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.615779\tvalid_0's auc: 0.756524\n",
      "[3]\tvalid_0's binary_logloss: 0.586994\tvalid_0's auc: 0.759714\n",
      "[4]\tvalid_0's binary_logloss: 0.563245\tvalid_0's auc: 0.761988\n",
      "[5]\tvalid_0's binary_logloss: 0.54326\tvalid_0's auc: 0.763567\n",
      "[6]\tvalid_0's binary_logloss: 0.526682\tvalid_0's auc: 0.762949\n",
      "[7]\tvalid_0's binary_logloss: 0.51248\tvalid_0's auc: 0.764754\n",
      "[8]\tvalid_0's binary_logloss: 0.500651\tvalid_0's auc: 0.764875\n",
      "[9]\tvalid_0's binary_logloss: 0.49096\tvalid_0's auc: 0.76325\n",
      "[10]\tvalid_0's binary_logloss: 0.482827\tvalid_0's auc: 0.763713\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[12]\tvalid_0's binary_logloss: 0.470064\tvalid_0's auc: 0.76424\n",
      "[13]\tvalid_0's binary_logloss: 0.464881\tvalid_0's auc: 0.765382\n",
      "[14]\tvalid_0's binary_logloss: 0.460645\tvalid_0's auc: 0.765338\n",
      "[15]\tvalid_0's binary_logloss: 0.457268\tvalid_0's auc: 0.765179\n",
      "[16]\tvalid_0's binary_logloss: 0.454458\tvalid_0's auc: 0.76512\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[1]\tvalid_0's binary_logloss: 0.650853\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.616135\tvalid_0's auc: 0.752078\n",
      "[3]\tvalid_0's binary_logloss: 0.587752\tvalid_0's auc: 0.755301\n",
      "[4]\tvalid_0's binary_logloss: 0.564477\tvalid_0's auc: 0.756622\n",
      "[5]\tvalid_0's binary_logloss: 0.544594\tvalid_0's auc: 0.760193\n",
      "[6]\tvalid_0's binary_logloss: 0.527785\tvalid_0's auc: 0.760906\n",
      "[7]\tvalid_0's binary_logloss: 0.513856\tvalid_0's auc: 0.761905\n",
      "[8]\tvalid_0's binary_logloss: 0.502076\tvalid_0's auc: 0.76343\n",
      "[9]\tvalid_0's binary_logloss: 0.491974\tvalid_0's auc: 0.763981\n",
      "[10]\tvalid_0's binary_logloss: 0.483799\tvalid_0's auc: 0.763893\n",
      "[11]\tvalid_0's binary_logloss: 0.476724\tvalid_0's auc: 0.765105\n",
      "[12]\tvalid_0's binary_logloss: 0.470592\tvalid_0's auc: 0.766134\n",
      "[13]\tvalid_0's binary_logloss: 0.465488\tvalid_0's auc: 0.766579\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[15]\tvalid_0's binary_logloss: 0.458058\tvalid_0's auc: 0.765453\n",
      "[16]\tvalid_0's binary_logloss: 0.455224\tvalid_0's auc: 0.765914\n",
      "[17]\tvalid_0's binary_logloss: 0.452883\tvalid_0's auc: 0.765398\n",
      "[18]\tvalid_0's binary_logloss: 0.450593\tvalid_0's auc: 0.766378\n",
      "[19]\tvalid_0's binary_logloss: 0.44907\tvalid_0's auc: 0.765987\n",
      "Early stopping, best iteration is:\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[1]\tvalid_0's binary_logloss: 0.650327\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.61564\tvalid_0's auc: 0.757175\n",
      "[3]\tvalid_0's binary_logloss: 0.586972\tvalid_0's auc: 0.764043\n",
      "[4]\tvalid_0's binary_logloss: 0.562638\tvalid_0's auc: 0.766654\n",
      "[5]\tvalid_0's binary_logloss: 0.542872\tvalid_0's auc: 0.766568\n",
      "[6]\tvalid_0's binary_logloss: 0.52609\tvalid_0's auc: 0.768887\n",
      "[7]\tvalid_0's binary_logloss: 0.512487\tvalid_0's auc: 0.769703\n",
      "[8]\tvalid_0's binary_logloss: 0.500321\tvalid_0's auc: 0.771338\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[10]\tvalid_0's binary_logloss: 0.481959\tvalid_0's auc: 0.770874\n",
      "[11]\tvalid_0's binary_logloss: 0.475167\tvalid_0's auc: 0.771372\n",
      "[12]\tvalid_0's binary_logloss: 0.46925\tvalid_0's auc: 0.770875\n",
      "[13]\tvalid_0's binary_logloss: 0.464184\tvalid_0's auc: 0.771062\n",
      "[14]\tvalid_0's binary_logloss: 0.460043\tvalid_0's auc: 0.770341\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[1]\tvalid_0's binary_logloss: 0.650309\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.615779\tvalid_0's auc: 0.756524\n",
      "[3]\tvalid_0's binary_logloss: 0.586994\tvalid_0's auc: 0.759714\n",
      "[4]\tvalid_0's binary_logloss: 0.563245\tvalid_0's auc: 0.761988\n",
      "[5]\tvalid_0's binary_logloss: 0.54326\tvalid_0's auc: 0.763567\n",
      "[6]\tvalid_0's binary_logloss: 0.526682\tvalid_0's auc: 0.762949\n",
      "[7]\tvalid_0's binary_logloss: 0.51248\tvalid_0's auc: 0.764754\n",
      "[8]\tvalid_0's binary_logloss: 0.500651\tvalid_0's auc: 0.764875\n",
      "[9]\tvalid_0's binary_logloss: 0.49096\tvalid_0's auc: 0.76325\n",
      "[10]\tvalid_0's binary_logloss: 0.482827\tvalid_0's auc: 0.763713\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[12]\tvalid_0's binary_logloss: 0.470064\tvalid_0's auc: 0.76424\n",
      "[13]\tvalid_0's binary_logloss: 0.464881\tvalid_0's auc: 0.765382\n",
      "[14]\tvalid_0's binary_logloss: 0.460645\tvalid_0's auc: 0.765338\n",
      "[15]\tvalid_0's binary_logloss: 0.457268\tvalid_0's auc: 0.765179\n",
      "[16]\tvalid_0's binary_logloss: 0.454458\tvalid_0's auc: 0.76512\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.650853\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.616135\tvalid_0's auc: 0.752078\n",
      "[3]\tvalid_0's binary_logloss: 0.587752\tvalid_0's auc: 0.755301\n",
      "[4]\tvalid_0's binary_logloss: 0.564477\tvalid_0's auc: 0.756622\n",
      "[5]\tvalid_0's binary_logloss: 0.544594\tvalid_0's auc: 0.760193\n",
      "[6]\tvalid_0's binary_logloss: 0.527785\tvalid_0's auc: 0.760906\n",
      "[7]\tvalid_0's binary_logloss: 0.513856\tvalid_0's auc: 0.761905\n",
      "[8]\tvalid_0's binary_logloss: 0.502076\tvalid_0's auc: 0.76343\n",
      "[9]\tvalid_0's binary_logloss: 0.491974\tvalid_0's auc: 0.763981\n",
      "[10]\tvalid_0's binary_logloss: 0.483799\tvalid_0's auc: 0.763893\n",
      "[11]\tvalid_0's binary_logloss: 0.476724\tvalid_0's auc: 0.765105\n",
      "[12]\tvalid_0's binary_logloss: 0.470592\tvalid_0's auc: 0.766134\n",
      "[13]\tvalid_0's binary_logloss: 0.465488\tvalid_0's auc: 0.766579\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[15]\tvalid_0's binary_logloss: 0.458058\tvalid_0's auc: 0.765453\n",
      "[16]\tvalid_0's binary_logloss: 0.455224\tvalid_0's auc: 0.765914\n",
      "[17]\tvalid_0's binary_logloss: 0.452883\tvalid_0's auc: 0.765398\n",
      "[18]\tvalid_0's binary_logloss: 0.450593\tvalid_0's auc: 0.766378\n",
      "[19]\tvalid_0's binary_logloss: 0.44907\tvalid_0's auc: 0.765987\n",
      "Early stopping, best iteration is:\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[1]\tvalid_0's binary_logloss: 0.650327\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.61564\tvalid_0's auc: 0.757175\n",
      "[3]\tvalid_0's binary_logloss: 0.586972\tvalid_0's auc: 0.764043\n",
      "[4]\tvalid_0's binary_logloss: 0.562638\tvalid_0's auc: 0.766654\n",
      "[5]\tvalid_0's binary_logloss: 0.542872\tvalid_0's auc: 0.766568\n",
      "[6]\tvalid_0's binary_logloss: 0.52609\tvalid_0's auc: 0.768887\n",
      "[7]\tvalid_0's binary_logloss: 0.512487\tvalid_0's auc: 0.769703\n",
      "[8]\tvalid_0's binary_logloss: 0.500321\tvalid_0's auc: 0.771338\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[10]\tvalid_0's binary_logloss: 0.481959\tvalid_0's auc: 0.770874\n",
      "[11]\tvalid_0's binary_logloss: 0.475167\tvalid_0's auc: 0.771372\n",
      "[12]\tvalid_0's binary_logloss: 0.46925\tvalid_0's auc: 0.770875\n",
      "[13]\tvalid_0's binary_logloss: 0.464184\tvalid_0's auc: 0.771062\n",
      "[14]\tvalid_0's binary_logloss: 0.460043\tvalid_0's auc: 0.770341\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[1]\tvalid_0's binary_logloss: 0.650309\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.615779\tvalid_0's auc: 0.756524\n",
      "[3]\tvalid_0's binary_logloss: 0.586994\tvalid_0's auc: 0.759714\n",
      "[4]\tvalid_0's binary_logloss: 0.563245\tvalid_0's auc: 0.761988\n",
      "[5]\tvalid_0's binary_logloss: 0.54326\tvalid_0's auc: 0.763567\n",
      "[6]\tvalid_0's binary_logloss: 0.526682\tvalid_0's auc: 0.762949\n",
      "[7]\tvalid_0's binary_logloss: 0.51248\tvalid_0's auc: 0.764754\n",
      "[8]\tvalid_0's binary_logloss: 0.500651\tvalid_0's auc: 0.764875\n",
      "[9]\tvalid_0's binary_logloss: 0.49096\tvalid_0's auc: 0.76325\n",
      "[10]\tvalid_0's binary_logloss: 0.482827\tvalid_0's auc: 0.763713\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[12]\tvalid_0's binary_logloss: 0.470064\tvalid_0's auc: 0.76424\n",
      "[13]\tvalid_0's binary_logloss: 0.464881\tvalid_0's auc: 0.765382\n",
      "[14]\tvalid_0's binary_logloss: 0.460645\tvalid_0's auc: 0.765338\n",
      "[15]\tvalid_0's binary_logloss: 0.457268\tvalid_0's auc: 0.765179\n",
      "[16]\tvalid_0's binary_logloss: 0.454458\tvalid_0's auc: 0.76512\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[1]\tvalid_0's binary_logloss: 0.650853\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.616135\tvalid_0's auc: 0.752078\n",
      "[3]\tvalid_0's binary_logloss: 0.587752\tvalid_0's auc: 0.755301\n",
      "[4]\tvalid_0's binary_logloss: 0.564477\tvalid_0's auc: 0.756622\n",
      "[5]\tvalid_0's binary_logloss: 0.544594\tvalid_0's auc: 0.760193\n",
      "[6]\tvalid_0's binary_logloss: 0.527785\tvalid_0's auc: 0.760906\n",
      "[7]\tvalid_0's binary_logloss: 0.513856\tvalid_0's auc: 0.761905\n",
      "[8]\tvalid_0's binary_logloss: 0.502076\tvalid_0's auc: 0.76343\n",
      "[9]\tvalid_0's binary_logloss: 0.491974\tvalid_0's auc: 0.763981\n",
      "[10]\tvalid_0's binary_logloss: 0.483799\tvalid_0's auc: 0.763893\n",
      "[11]\tvalid_0's binary_logloss: 0.476724\tvalid_0's auc: 0.765105\n",
      "[12]\tvalid_0's binary_logloss: 0.470592\tvalid_0's auc: 0.766134\n",
      "[13]\tvalid_0's binary_logloss: 0.465488\tvalid_0's auc: 0.766579\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[15]\tvalid_0's binary_logloss: 0.458058\tvalid_0's auc: 0.765453\n",
      "[16]\tvalid_0's binary_logloss: 0.455224\tvalid_0's auc: 0.765914\n",
      "[17]\tvalid_0's binary_logloss: 0.452883\tvalid_0's auc: 0.765398\n",
      "[18]\tvalid_0's binary_logloss: 0.450593\tvalid_0's auc: 0.766378\n",
      "[19]\tvalid_0's binary_logloss: 0.44907\tvalid_0's auc: 0.765987\n",
      "Early stopping, best iteration is:\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[1]\tvalid_0's binary_logloss: 0.650327\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.61564\tvalid_0's auc: 0.757175\n",
      "[3]\tvalid_0's binary_logloss: 0.586972\tvalid_0's auc: 0.764043\n",
      "[4]\tvalid_0's binary_logloss: 0.562638\tvalid_0's auc: 0.766654\n",
      "[5]\tvalid_0's binary_logloss: 0.542872\tvalid_0's auc: 0.766568\n",
      "[6]\tvalid_0's binary_logloss: 0.52609\tvalid_0's auc: 0.768887\n",
      "[7]\tvalid_0's binary_logloss: 0.512487\tvalid_0's auc: 0.769703\n",
      "[8]\tvalid_0's binary_logloss: 0.500321\tvalid_0's auc: 0.771338\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[10]\tvalid_0's binary_logloss: 0.481959\tvalid_0's auc: 0.770874\n",
      "[11]\tvalid_0's binary_logloss: 0.475167\tvalid_0's auc: 0.771372\n",
      "[12]\tvalid_0's binary_logloss: 0.46925\tvalid_0's auc: 0.770875\n",
      "[13]\tvalid_0's binary_logloss: 0.464184\tvalid_0's auc: 0.771062\n",
      "[14]\tvalid_0's binary_logloss: 0.460043\tvalid_0's auc: 0.770341\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[1]\tvalid_0's binary_logloss: 0.650309\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.615779\tvalid_0's auc: 0.756524\n",
      "[3]\tvalid_0's binary_logloss: 0.586994\tvalid_0's auc: 0.759714\n",
      "[4]\tvalid_0's binary_logloss: 0.563245\tvalid_0's auc: 0.761988\n",
      "[5]\tvalid_0's binary_logloss: 0.54326\tvalid_0's auc: 0.763567\n",
      "[6]\tvalid_0's binary_logloss: 0.526682\tvalid_0's auc: 0.762949\n",
      "[7]\tvalid_0's binary_logloss: 0.51248\tvalid_0's auc: 0.764754\n",
      "[8]\tvalid_0's binary_logloss: 0.500651\tvalid_0's auc: 0.764875\n",
      "[9]\tvalid_0's binary_logloss: 0.49096\tvalid_0's auc: 0.76325\n",
      "[10]\tvalid_0's binary_logloss: 0.482827\tvalid_0's auc: 0.763713\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[12]\tvalid_0's binary_logloss: 0.470064\tvalid_0's auc: 0.76424\n",
      "[13]\tvalid_0's binary_logloss: 0.464881\tvalid_0's auc: 0.765382\n",
      "[14]\tvalid_0's binary_logloss: 0.460645\tvalid_0's auc: 0.765338\n",
      "[15]\tvalid_0's binary_logloss: 0.457268\tvalid_0's auc: 0.765179\n",
      "[16]\tvalid_0's binary_logloss: 0.454458\tvalid_0's auc: 0.76512\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[1]\tvalid_0's binary_logloss: 0.650853\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.616135\tvalid_0's auc: 0.752078\n",
      "[3]\tvalid_0's binary_logloss: 0.587752\tvalid_0's auc: 0.755301\n",
      "[4]\tvalid_0's binary_logloss: 0.564477\tvalid_0's auc: 0.756622\n",
      "[5]\tvalid_0's binary_logloss: 0.544594\tvalid_0's auc: 0.760193\n",
      "[6]\tvalid_0's binary_logloss: 0.527785\tvalid_0's auc: 0.760906\n",
      "[7]\tvalid_0's binary_logloss: 0.513856\tvalid_0's auc: 0.761905\n",
      "[8]\tvalid_0's binary_logloss: 0.502076\tvalid_0's auc: 0.76343\n",
      "[9]\tvalid_0's binary_logloss: 0.491974\tvalid_0's auc: 0.763981\n",
      "[10]\tvalid_0's binary_logloss: 0.483799\tvalid_0's auc: 0.763893\n",
      "[11]\tvalid_0's binary_logloss: 0.476724\tvalid_0's auc: 0.765105\n",
      "[12]\tvalid_0's binary_logloss: 0.470592\tvalid_0's auc: 0.766134\n",
      "[13]\tvalid_0's binary_logloss: 0.465488\tvalid_0's auc: 0.766579\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[15]\tvalid_0's binary_logloss: 0.458058\tvalid_0's auc: 0.765453\n",
      "[16]\tvalid_0's binary_logloss: 0.455224\tvalid_0's auc: 0.765914\n",
      "[17]\tvalid_0's binary_logloss: 0.452883\tvalid_0's auc: 0.765398\n",
      "[18]\tvalid_0's binary_logloss: 0.450593\tvalid_0's auc: 0.766378\n",
      "[19]\tvalid_0's binary_logloss: 0.44907\tvalid_0's auc: 0.765987\n",
      "Early stopping, best iteration is:\n",
      "[14]\tvalid_0's binary_logloss: 0.4612\tvalid_0's auc: 0.766906\n",
      "[1]\tvalid_0's binary_logloss: 0.650327\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.61564\tvalid_0's auc: 0.757175\n",
      "[3]\tvalid_0's binary_logloss: 0.586972\tvalid_0's auc: 0.764043\n",
      "[4]\tvalid_0's binary_logloss: 0.562638\tvalid_0's auc: 0.766654\n",
      "[5]\tvalid_0's binary_logloss: 0.542872\tvalid_0's auc: 0.766568\n",
      "[6]\tvalid_0's binary_logloss: 0.52609\tvalid_0's auc: 0.768887\n",
      "[7]\tvalid_0's binary_logloss: 0.512487\tvalid_0's auc: 0.769703\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8]\tvalid_0's binary_logloss: 0.500321\tvalid_0's auc: 0.771338\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[10]\tvalid_0's binary_logloss: 0.481959\tvalid_0's auc: 0.770874\n",
      "[11]\tvalid_0's binary_logloss: 0.475167\tvalid_0's auc: 0.771372\n",
      "[12]\tvalid_0's binary_logloss: 0.46925\tvalid_0's auc: 0.770875\n",
      "[13]\tvalid_0's binary_logloss: 0.464184\tvalid_0's auc: 0.771062\n",
      "[14]\tvalid_0's binary_logloss: 0.460043\tvalid_0's auc: 0.770341\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.490032\tvalid_0's auc: 0.772365\n",
      "[1]\tvalid_0's binary_logloss: 0.650309\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.615779\tvalid_0's auc: 0.756524\n",
      "[3]\tvalid_0's binary_logloss: 0.586994\tvalid_0's auc: 0.759714\n",
      "[4]\tvalid_0's binary_logloss: 0.563245\tvalid_0's auc: 0.761988\n",
      "[5]\tvalid_0's binary_logloss: 0.54326\tvalid_0's auc: 0.763567\n",
      "[6]\tvalid_0's binary_logloss: 0.526682\tvalid_0's auc: 0.762949\n",
      "[7]\tvalid_0's binary_logloss: 0.51248\tvalid_0's auc: 0.764754\n",
      "[8]\tvalid_0's binary_logloss: 0.500651\tvalid_0's auc: 0.764875\n",
      "[9]\tvalid_0's binary_logloss: 0.49096\tvalid_0's auc: 0.76325\n",
      "[10]\tvalid_0's binary_logloss: 0.482827\tvalid_0's auc: 0.763713\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[12]\tvalid_0's binary_logloss: 0.470064\tvalid_0's auc: 0.76424\n",
      "[13]\tvalid_0's binary_logloss: 0.464881\tvalid_0's auc: 0.765382\n",
      "[14]\tvalid_0's binary_logloss: 0.460645\tvalid_0's auc: 0.765338\n",
      "[15]\tvalid_0's binary_logloss: 0.457268\tvalid_0's auc: 0.765179\n",
      "[16]\tvalid_0's binary_logloss: 0.454458\tvalid_0's auc: 0.76512\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.475711\tvalid_0's auc: 0.765427\n",
      "[1]\tvalid_0's binary_logloss: 0.641006\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.60046\tvalid_0's auc: 0.751714\n",
      "[3]\tvalid_0's binary_logloss: 0.568982\tvalid_0's auc: 0.755611\n",
      "[4]\tvalid_0's binary_logloss: 0.543679\tvalid_0's auc: 0.75906\n",
      "[5]\tvalid_0's binary_logloss: 0.523356\tvalid_0's auc: 0.762148\n",
      "[6]\tvalid_0's binary_logloss: 0.507119\tvalid_0's auc: 0.760809\n",
      "[7]\tvalid_0's binary_logloss: 0.493783\tvalid_0's auc: 0.763779\n",
      "[8]\tvalid_0's binary_logloss: 0.482963\tvalid_0's auc: 0.764698\n",
      "[9]\tvalid_0's binary_logloss: 0.474584\tvalid_0's auc: 0.765261\n",
      "[10]\tvalid_0's binary_logloss: 0.467761\tvalid_0's auc: 0.76567\n",
      "[11]\tvalid_0's binary_logloss: 0.462211\tvalid_0's auc: 0.765548\n",
      "[12]\tvalid_0's binary_logloss: 0.457512\tvalid_0's auc: 0.766293\n",
      "[13]\tvalid_0's binary_logloss: 0.453961\tvalid_0's auc: 0.766222\n",
      "[14]\tvalid_0's binary_logloss: 0.451213\tvalid_0's auc: 0.766377\n",
      "[15]\tvalid_0's binary_logloss: 0.449126\tvalid_0's auc: 0.766314\n",
      "[16]\tvalid_0's binary_logloss: 0.447457\tvalid_0's auc: 0.765742\n",
      "[17]\tvalid_0's binary_logloss: 0.445718\tvalid_0's auc: 0.766506\n",
      "[18]\tvalid_0's binary_logloss: 0.444613\tvalid_0's auc: 0.766507\n",
      "[19]\tvalid_0's binary_logloss: 0.443095\tvalid_0's auc: 0.768662\n",
      "[20]\tvalid_0's binary_logloss: 0.442581\tvalid_0's auc: 0.768199\n",
      "[21]\tvalid_0's binary_logloss: 0.441855\tvalid_0's auc: 0.768631\n",
      "[22]\tvalid_0's binary_logloss: 0.441239\tvalid_0's auc: 0.769082\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[24]\tvalid_0's binary_logloss: 0.441121\tvalid_0's auc: 0.768284\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[1]\tvalid_0's binary_logloss: 0.640346\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.599816\tvalid_0's auc: 0.756783\n",
      "[3]\tvalid_0's binary_logloss: 0.567876\tvalid_0's auc: 0.762467\n",
      "[4]\tvalid_0's binary_logloss: 0.542429\tvalid_0's auc: 0.763728\n",
      "[5]\tvalid_0's binary_logloss: 0.522117\tvalid_0's auc: 0.765465\n",
      "[6]\tvalid_0's binary_logloss: 0.506238\tvalid_0's auc: 0.765642\n",
      "[7]\tvalid_0's binary_logloss: 0.493102\tvalid_0's auc: 0.765213\n",
      "[8]\tvalid_0's binary_logloss: 0.481931\tvalid_0's auc: 0.768695\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[10]\tvalid_0's binary_logloss: 0.466103\tvalid_0's auc: 0.769675\n",
      "[11]\tvalid_0's binary_logloss: 0.460544\tvalid_0's auc: 0.769332\n",
      "[12]\tvalid_0's binary_logloss: 0.456187\tvalid_0's auc: 0.769133\n",
      "[13]\tvalid_0's binary_logloss: 0.452927\tvalid_0's auc: 0.76845\n",
      "[14]\tvalid_0's binary_logloss: 0.44965\tvalid_0's auc: 0.769096\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[1]\tvalid_0's binary_logloss: 0.640322\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.600099\tvalid_0's auc: 0.756076\n",
      "[3]\tvalid_0's binary_logloss: 0.567856\tvalid_0's auc: 0.761172\n",
      "[4]\tvalid_0's binary_logloss: 0.542555\tvalid_0's auc: 0.764108\n",
      "[5]\tvalid_0's binary_logloss: 0.522033\tvalid_0's auc: 0.765739\n",
      "[6]\tvalid_0's binary_logloss: 0.505883\tvalid_0's auc: 0.764746\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[8]\tvalid_0's binary_logloss: 0.482527\tvalid_0's auc: 0.763353\n",
      "[9]\tvalid_0's binary_logloss: 0.474211\tvalid_0's auc: 0.76342\n",
      "[10]\tvalid_0's binary_logloss: 0.467219\tvalid_0's auc: 0.763589\n",
      "[11]\tvalid_0's binary_logloss: 0.461432\tvalid_0's auc: 0.765676\n",
      "[12]\tvalid_0's binary_logloss: 0.456908\tvalid_0's auc: 0.766021\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[1]\tvalid_0's binary_logloss: 0.641006\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.60046\tvalid_0's auc: 0.751714\n",
      "[3]\tvalid_0's binary_logloss: 0.568982\tvalid_0's auc: 0.755611\n",
      "[4]\tvalid_0's binary_logloss: 0.543679\tvalid_0's auc: 0.75906\n",
      "[5]\tvalid_0's binary_logloss: 0.523356\tvalid_0's auc: 0.762148\n",
      "[6]\tvalid_0's binary_logloss: 0.507119\tvalid_0's auc: 0.760809\n",
      "[7]\tvalid_0's binary_logloss: 0.493783\tvalid_0's auc: 0.763779\n",
      "[8]\tvalid_0's binary_logloss: 0.482963\tvalid_0's auc: 0.764698\n",
      "[9]\tvalid_0's binary_logloss: 0.474584\tvalid_0's auc: 0.765261\n",
      "[10]\tvalid_0's binary_logloss: 0.467761\tvalid_0's auc: 0.76567\n",
      "[11]\tvalid_0's binary_logloss: 0.462211\tvalid_0's auc: 0.765548\n",
      "[12]\tvalid_0's binary_logloss: 0.457512\tvalid_0's auc: 0.766293\n",
      "[13]\tvalid_0's binary_logloss: 0.453961\tvalid_0's auc: 0.766222\n",
      "[14]\tvalid_0's binary_logloss: 0.451213\tvalid_0's auc: 0.766377\n",
      "[15]\tvalid_0's binary_logloss: 0.449126\tvalid_0's auc: 0.766314\n",
      "[16]\tvalid_0's binary_logloss: 0.447457\tvalid_0's auc: 0.765742\n",
      "[17]\tvalid_0's binary_logloss: 0.445718\tvalid_0's auc: 0.766506\n",
      "[18]\tvalid_0's binary_logloss: 0.444613\tvalid_0's auc: 0.766507\n",
      "[19]\tvalid_0's binary_logloss: 0.443095\tvalid_0's auc: 0.768662\n",
      "[20]\tvalid_0's binary_logloss: 0.442581\tvalid_0's auc: 0.768199\n",
      "[21]\tvalid_0's binary_logloss: 0.441855\tvalid_0's auc: 0.768631\n",
      "[22]\tvalid_0's binary_logloss: 0.441239\tvalid_0's auc: 0.769082\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[24]\tvalid_0's binary_logloss: 0.441121\tvalid_0's auc: 0.768284\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[26]\tvalid_0's binary_logloss: 0.440982\tvalid_0's auc: 0.767638\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[1]\tvalid_0's binary_logloss: 0.640346\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.599816\tvalid_0's auc: 0.756783\n",
      "[3]\tvalid_0's binary_logloss: 0.567876\tvalid_0's auc: 0.762467\n",
      "[4]\tvalid_0's binary_logloss: 0.542429\tvalid_0's auc: 0.763728\n",
      "[5]\tvalid_0's binary_logloss: 0.522117\tvalid_0's auc: 0.765465\n",
      "[6]\tvalid_0's binary_logloss: 0.506238\tvalid_0's auc: 0.765642\n",
      "[7]\tvalid_0's binary_logloss: 0.493102\tvalid_0's auc: 0.765213\n",
      "[8]\tvalid_0's binary_logloss: 0.481931\tvalid_0's auc: 0.768695\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[10]\tvalid_0's binary_logloss: 0.466103\tvalid_0's auc: 0.769675\n",
      "[11]\tvalid_0's binary_logloss: 0.460544\tvalid_0's auc: 0.769332\n",
      "[12]\tvalid_0's binary_logloss: 0.456187\tvalid_0's auc: 0.769133\n",
      "[13]\tvalid_0's binary_logloss: 0.452927\tvalid_0's auc: 0.76845\n",
      "[14]\tvalid_0's binary_logloss: 0.44965\tvalid_0's auc: 0.769096\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[1]\tvalid_0's binary_logloss: 0.640322\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.600099\tvalid_0's auc: 0.756076\n",
      "[3]\tvalid_0's binary_logloss: 0.567856\tvalid_0's auc: 0.761172\n",
      "[4]\tvalid_0's binary_logloss: 0.542555\tvalid_0's auc: 0.764108\n",
      "[5]\tvalid_0's binary_logloss: 0.522033\tvalid_0's auc: 0.765739\n",
      "[6]\tvalid_0's binary_logloss: 0.505883\tvalid_0's auc: 0.764746\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[8]\tvalid_0's binary_logloss: 0.482527\tvalid_0's auc: 0.763353\n",
      "[9]\tvalid_0's binary_logloss: 0.474211\tvalid_0's auc: 0.76342\n",
      "[10]\tvalid_0's binary_logloss: 0.467219\tvalid_0's auc: 0.763589\n",
      "[11]\tvalid_0's binary_logloss: 0.461432\tvalid_0's auc: 0.765676\n",
      "[12]\tvalid_0's binary_logloss: 0.456908\tvalid_0's auc: 0.766021\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[1]\tvalid_0's binary_logloss: 0.641006\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.60046\tvalid_0's auc: 0.751714\n",
      "[3]\tvalid_0's binary_logloss: 0.568982\tvalid_0's auc: 0.755611\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4]\tvalid_0's binary_logloss: 0.543679\tvalid_0's auc: 0.75906\n",
      "[5]\tvalid_0's binary_logloss: 0.523356\tvalid_0's auc: 0.762148\n",
      "[6]\tvalid_0's binary_logloss: 0.507119\tvalid_0's auc: 0.760809\n",
      "[7]\tvalid_0's binary_logloss: 0.493783\tvalid_0's auc: 0.763779\n",
      "[8]\tvalid_0's binary_logloss: 0.482963\tvalid_0's auc: 0.764698\n",
      "[9]\tvalid_0's binary_logloss: 0.474584\tvalid_0's auc: 0.765261\n",
      "[10]\tvalid_0's binary_logloss: 0.467761\tvalid_0's auc: 0.76567\n",
      "[11]\tvalid_0's binary_logloss: 0.462211\tvalid_0's auc: 0.765548\n",
      "[12]\tvalid_0's binary_logloss: 0.457512\tvalid_0's auc: 0.766293\n",
      "[13]\tvalid_0's binary_logloss: 0.453961\tvalid_0's auc: 0.766222\n",
      "[14]\tvalid_0's binary_logloss: 0.451213\tvalid_0's auc: 0.766377\n",
      "[15]\tvalid_0's binary_logloss: 0.449126\tvalid_0's auc: 0.766314\n",
      "[16]\tvalid_0's binary_logloss: 0.447457\tvalid_0's auc: 0.765742\n",
      "[17]\tvalid_0's binary_logloss: 0.445718\tvalid_0's auc: 0.766506\n",
      "[18]\tvalid_0's binary_logloss: 0.444613\tvalid_0's auc: 0.766507\n",
      "[19]\tvalid_0's binary_logloss: 0.443095\tvalid_0's auc: 0.768662\n",
      "[20]\tvalid_0's binary_logloss: 0.442581\tvalid_0's auc: 0.768199\n",
      "[21]\tvalid_0's binary_logloss: 0.441855\tvalid_0's auc: 0.768631\n",
      "[22]\tvalid_0's binary_logloss: 0.441239\tvalid_0's auc: 0.769082\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[24]\tvalid_0's binary_logloss: 0.441121\tvalid_0's auc: 0.768284\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[26]\tvalid_0's binary_logloss: 0.440982\tvalid_0's auc: 0.767638\n",
      "[27]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.766989\n",
      "[28]\tvalid_0's binary_logloss: 0.441096\tvalid_0's auc: 0.766995\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[1]\tvalid_0's binary_logloss: 0.640346\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.599816\tvalid_0's auc: 0.756783\n",
      "[3]\tvalid_0's binary_logloss: 0.567876\tvalid_0's auc: 0.762467\n",
      "[4]\tvalid_0's binary_logloss: 0.542429\tvalid_0's auc: 0.763728\n",
      "[5]\tvalid_0's binary_logloss: 0.522117\tvalid_0's auc: 0.765465\n",
      "[6]\tvalid_0's binary_logloss: 0.506238\tvalid_0's auc: 0.765642\n",
      "[7]\tvalid_0's binary_logloss: 0.493102\tvalid_0's auc: 0.765213\n",
      "[8]\tvalid_0's binary_logloss: 0.481931\tvalid_0's auc: 0.768695\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[10]\tvalid_0's binary_logloss: 0.466103\tvalid_0's auc: 0.769675\n",
      "[11]\tvalid_0's binary_logloss: 0.460544\tvalid_0's auc: 0.769332\n",
      "[12]\tvalid_0's binary_logloss: 0.456187\tvalid_0's auc: 0.769133\n",
      "[13]\tvalid_0's binary_logloss: 0.452927\tvalid_0's auc: 0.76845\n",
      "[14]\tvalid_0's binary_logloss: 0.44965\tvalid_0's auc: 0.769096\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[1]\tvalid_0's binary_logloss: 0.640322\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.600099\tvalid_0's auc: 0.756076\n",
      "[3]\tvalid_0's binary_logloss: 0.567856\tvalid_0's auc: 0.761172\n",
      "[4]\tvalid_0's binary_logloss: 0.542555\tvalid_0's auc: 0.764108\n",
      "[5]\tvalid_0's binary_logloss: 0.522033\tvalid_0's auc: 0.765739\n",
      "[6]\tvalid_0's binary_logloss: 0.505883\tvalid_0's auc: 0.764746\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[8]\tvalid_0's binary_logloss: 0.482527\tvalid_0's auc: 0.763353\n",
      "[9]\tvalid_0's binary_logloss: 0.474211\tvalid_0's auc: 0.76342\n",
      "[10]\tvalid_0's binary_logloss: 0.467219\tvalid_0's auc: 0.763589\n",
      "[11]\tvalid_0's binary_logloss: 0.461432\tvalid_0's auc: 0.765676\n",
      "[12]\tvalid_0's binary_logloss: 0.456908\tvalid_0's auc: 0.766021\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[1]\tvalid_0's binary_logloss: 0.641006\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.60046\tvalid_0's auc: 0.751714\n",
      "[3]\tvalid_0's binary_logloss: 0.568982\tvalid_0's auc: 0.755611\n",
      "[4]\tvalid_0's binary_logloss: 0.543679\tvalid_0's auc: 0.75906\n",
      "[5]\tvalid_0's binary_logloss: 0.523356\tvalid_0's auc: 0.762148\n",
      "[6]\tvalid_0's binary_logloss: 0.507119\tvalid_0's auc: 0.760809\n",
      "[7]\tvalid_0's binary_logloss: 0.493783\tvalid_0's auc: 0.763779\n",
      "[8]\tvalid_0's binary_logloss: 0.482963\tvalid_0's auc: 0.764698\n",
      "[9]\tvalid_0's binary_logloss: 0.474584\tvalid_0's auc: 0.765261\n",
      "[10]\tvalid_0's binary_logloss: 0.467761\tvalid_0's auc: 0.76567\n",
      "[11]\tvalid_0's binary_logloss: 0.462211\tvalid_0's auc: 0.765548\n",
      "[12]\tvalid_0's binary_logloss: 0.457512\tvalid_0's auc: 0.766293\n",
      "[13]\tvalid_0's binary_logloss: 0.453961\tvalid_0's auc: 0.766222\n",
      "[14]\tvalid_0's binary_logloss: 0.451213\tvalid_0's auc: 0.766377\n",
      "[15]\tvalid_0's binary_logloss: 0.449126\tvalid_0's auc: 0.766314\n",
      "[16]\tvalid_0's binary_logloss: 0.447457\tvalid_0's auc: 0.765742\n",
      "[17]\tvalid_0's binary_logloss: 0.445718\tvalid_0's auc: 0.766506\n",
      "[18]\tvalid_0's binary_logloss: 0.444613\tvalid_0's auc: 0.766507\n",
      "[19]\tvalid_0's binary_logloss: 0.443095\tvalid_0's auc: 0.768662\n",
      "[20]\tvalid_0's binary_logloss: 0.442581\tvalid_0's auc: 0.768199\n",
      "[21]\tvalid_0's binary_logloss: 0.441855\tvalid_0's auc: 0.768631\n",
      "[22]\tvalid_0's binary_logloss: 0.441239\tvalid_0's auc: 0.769082\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[24]\tvalid_0's binary_logloss: 0.441121\tvalid_0's auc: 0.768284\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[26]\tvalid_0's binary_logloss: 0.440982\tvalid_0's auc: 0.767638\n",
      "[27]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.766989\n",
      "[28]\tvalid_0's binary_logloss: 0.441096\tvalid_0's auc: 0.766995\n",
      "Early stopping, best iteration is:\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[1]\tvalid_0's binary_logloss: 0.640346\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.599816\tvalid_0's auc: 0.756783\n",
      "[3]\tvalid_0's binary_logloss: 0.567876\tvalid_0's auc: 0.762467\n",
      "[4]\tvalid_0's binary_logloss: 0.542429\tvalid_0's auc: 0.763728\n",
      "[5]\tvalid_0's binary_logloss: 0.522117\tvalid_0's auc: 0.765465\n",
      "[6]\tvalid_0's binary_logloss: 0.506238\tvalid_0's auc: 0.765642\n",
      "[7]\tvalid_0's binary_logloss: 0.493102\tvalid_0's auc: 0.765213\n",
      "[8]\tvalid_0's binary_logloss: 0.481931\tvalid_0's auc: 0.768695\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[10]\tvalid_0's binary_logloss: 0.466103\tvalid_0's auc: 0.769675\n",
      "[11]\tvalid_0's binary_logloss: 0.460544\tvalid_0's auc: 0.769332\n",
      "[12]\tvalid_0's binary_logloss: 0.456187\tvalid_0's auc: 0.769133\n",
      "[13]\tvalid_0's binary_logloss: 0.452927\tvalid_0's auc: 0.76845\n",
      "[14]\tvalid_0's binary_logloss: 0.44965\tvalid_0's auc: 0.769096\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[1]\tvalid_0's binary_logloss: 0.640322\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.600099\tvalid_0's auc: 0.756076\n",
      "[3]\tvalid_0's binary_logloss: 0.567856\tvalid_0's auc: 0.761172\n",
      "[4]\tvalid_0's binary_logloss: 0.542555\tvalid_0's auc: 0.764108\n",
      "[5]\tvalid_0's binary_logloss: 0.522033\tvalid_0's auc: 0.765739\n",
      "[6]\tvalid_0's binary_logloss: 0.505883\tvalid_0's auc: 0.764746\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[8]\tvalid_0's binary_logloss: 0.482527\tvalid_0's auc: 0.763353\n",
      "[9]\tvalid_0's binary_logloss: 0.474211\tvalid_0's auc: 0.76342\n",
      "[10]\tvalid_0's binary_logloss: 0.467219\tvalid_0's auc: 0.763589\n",
      "[11]\tvalid_0's binary_logloss: 0.461432\tvalid_0's auc: 0.765676\n",
      "[12]\tvalid_0's binary_logloss: 0.456908\tvalid_0's auc: 0.766021\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[1]\tvalid_0's binary_logloss: 0.641006\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.60046\tvalid_0's auc: 0.751714\n",
      "[3]\tvalid_0's binary_logloss: 0.568982\tvalid_0's auc: 0.755611\n",
      "[4]\tvalid_0's binary_logloss: 0.543679\tvalid_0's auc: 0.75906\n",
      "[5]\tvalid_0's binary_logloss: 0.523356\tvalid_0's auc: 0.762148\n",
      "[6]\tvalid_0's binary_logloss: 0.507119\tvalid_0's auc: 0.760809\n",
      "[7]\tvalid_0's binary_logloss: 0.493783\tvalid_0's auc: 0.763779\n",
      "[8]\tvalid_0's binary_logloss: 0.482963\tvalid_0's auc: 0.764698\n",
      "[9]\tvalid_0's binary_logloss: 0.474584\tvalid_0's auc: 0.765261\n",
      "[10]\tvalid_0's binary_logloss: 0.467761\tvalid_0's auc: 0.76567\n",
      "[11]\tvalid_0's binary_logloss: 0.462211\tvalid_0's auc: 0.765548\n",
      "[12]\tvalid_0's binary_logloss: 0.457512\tvalid_0's auc: 0.766293\n",
      "[13]\tvalid_0's binary_logloss: 0.453961\tvalid_0's auc: 0.766222\n",
      "[14]\tvalid_0's binary_logloss: 0.451213\tvalid_0's auc: 0.766377\n",
      "[15]\tvalid_0's binary_logloss: 0.449126\tvalid_0's auc: 0.766314\n",
      "[16]\tvalid_0's binary_logloss: 0.447457\tvalid_0's auc: 0.765742\n",
      "[17]\tvalid_0's binary_logloss: 0.445718\tvalid_0's auc: 0.766506\n",
      "[18]\tvalid_0's binary_logloss: 0.444613\tvalid_0's auc: 0.766507\n",
      "[19]\tvalid_0's binary_logloss: 0.443095\tvalid_0's auc: 0.768662\n",
      "[20]\tvalid_0's binary_logloss: 0.442581\tvalid_0's auc: 0.768199\n",
      "[21]\tvalid_0's binary_logloss: 0.441855\tvalid_0's auc: 0.768631\n",
      "[22]\tvalid_0's binary_logloss: 0.441239\tvalid_0's auc: 0.769082\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[24]\tvalid_0's binary_logloss: 0.441121\tvalid_0's auc: 0.768284\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[26]\tvalid_0's binary_logloss: 0.440982\tvalid_0's auc: 0.767638\n",
      "[27]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.766989\n",
      "[28]\tvalid_0's binary_logloss: 0.441096\tvalid_0's auc: 0.766995\n",
      "Early stopping, best iteration is:\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.640346\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.599816\tvalid_0's auc: 0.756783\n",
      "[3]\tvalid_0's binary_logloss: 0.567876\tvalid_0's auc: 0.762467\n",
      "[4]\tvalid_0's binary_logloss: 0.542429\tvalid_0's auc: 0.763728\n",
      "[5]\tvalid_0's binary_logloss: 0.522117\tvalid_0's auc: 0.765465\n",
      "[6]\tvalid_0's binary_logloss: 0.506238\tvalid_0's auc: 0.765642\n",
      "[7]\tvalid_0's binary_logloss: 0.493102\tvalid_0's auc: 0.765213\n",
      "[8]\tvalid_0's binary_logloss: 0.481931\tvalid_0's auc: 0.768695\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[10]\tvalid_0's binary_logloss: 0.466103\tvalid_0's auc: 0.769675\n",
      "[11]\tvalid_0's binary_logloss: 0.460544\tvalid_0's auc: 0.769332\n",
      "[12]\tvalid_0's binary_logloss: 0.456187\tvalid_0's auc: 0.769133\n",
      "[13]\tvalid_0's binary_logloss: 0.452927\tvalid_0's auc: 0.76845\n",
      "[14]\tvalid_0's binary_logloss: 0.44965\tvalid_0's auc: 0.769096\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[1]\tvalid_0's binary_logloss: 0.640322\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.600099\tvalid_0's auc: 0.756076\n",
      "[3]\tvalid_0's binary_logloss: 0.567856\tvalid_0's auc: 0.761172\n",
      "[4]\tvalid_0's binary_logloss: 0.542555\tvalid_0's auc: 0.764108\n",
      "[5]\tvalid_0's binary_logloss: 0.522033\tvalid_0's auc: 0.765739\n",
      "[6]\tvalid_0's binary_logloss: 0.505883\tvalid_0's auc: 0.764746\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[8]\tvalid_0's binary_logloss: 0.482527\tvalid_0's auc: 0.763353\n",
      "[9]\tvalid_0's binary_logloss: 0.474211\tvalid_0's auc: 0.76342\n",
      "[10]\tvalid_0's binary_logloss: 0.467219\tvalid_0's auc: 0.763589\n",
      "[11]\tvalid_0's binary_logloss: 0.461432\tvalid_0's auc: 0.765676\n",
      "[12]\tvalid_0's binary_logloss: 0.456908\tvalid_0's auc: 0.766021\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[1]\tvalid_0's binary_logloss: 0.641006\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.60046\tvalid_0's auc: 0.751714\n",
      "[3]\tvalid_0's binary_logloss: 0.568982\tvalid_0's auc: 0.755611\n",
      "[4]\tvalid_0's binary_logloss: 0.543679\tvalid_0's auc: 0.75906\n",
      "[5]\tvalid_0's binary_logloss: 0.523356\tvalid_0's auc: 0.762148\n",
      "[6]\tvalid_0's binary_logloss: 0.507119\tvalid_0's auc: 0.760809\n",
      "[7]\tvalid_0's binary_logloss: 0.493783\tvalid_0's auc: 0.763779\n",
      "[8]\tvalid_0's binary_logloss: 0.482963\tvalid_0's auc: 0.764698\n",
      "[9]\tvalid_0's binary_logloss: 0.474584\tvalid_0's auc: 0.765261\n",
      "[10]\tvalid_0's binary_logloss: 0.467761\tvalid_0's auc: 0.76567\n",
      "[11]\tvalid_0's binary_logloss: 0.462211\tvalid_0's auc: 0.765548\n",
      "[12]\tvalid_0's binary_logloss: 0.457512\tvalid_0's auc: 0.766293\n",
      "[13]\tvalid_0's binary_logloss: 0.453961\tvalid_0's auc: 0.766222\n",
      "[14]\tvalid_0's binary_logloss: 0.451213\tvalid_0's auc: 0.766377\n",
      "[15]\tvalid_0's binary_logloss: 0.449126\tvalid_0's auc: 0.766314\n",
      "[16]\tvalid_0's binary_logloss: 0.447457\tvalid_0's auc: 0.765742\n",
      "[17]\tvalid_0's binary_logloss: 0.445718\tvalid_0's auc: 0.766506\n",
      "[18]\tvalid_0's binary_logloss: 0.444613\tvalid_0's auc: 0.766507\n",
      "[19]\tvalid_0's binary_logloss: 0.443095\tvalid_0's auc: 0.768662\n",
      "[20]\tvalid_0's binary_logloss: 0.442581\tvalid_0's auc: 0.768199\n",
      "[21]\tvalid_0's binary_logloss: 0.441855\tvalid_0's auc: 0.768631\n",
      "[22]\tvalid_0's binary_logloss: 0.441239\tvalid_0's auc: 0.769082\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[24]\tvalid_0's binary_logloss: 0.441121\tvalid_0's auc: 0.768284\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[26]\tvalid_0's binary_logloss: 0.440982\tvalid_0's auc: 0.767638\n",
      "[27]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.766989\n",
      "[28]\tvalid_0's binary_logloss: 0.441096\tvalid_0's auc: 0.766995\n",
      "Early stopping, best iteration is:\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[1]\tvalid_0's binary_logloss: 0.640346\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.599816\tvalid_0's auc: 0.756783\n",
      "[3]\tvalid_0's binary_logloss: 0.567876\tvalid_0's auc: 0.762467\n",
      "[4]\tvalid_0's binary_logloss: 0.542429\tvalid_0's auc: 0.763728\n",
      "[5]\tvalid_0's binary_logloss: 0.522117\tvalid_0's auc: 0.765465\n",
      "[6]\tvalid_0's binary_logloss: 0.506238\tvalid_0's auc: 0.765642\n",
      "[7]\tvalid_0's binary_logloss: 0.493102\tvalid_0's auc: 0.765213\n",
      "[8]\tvalid_0's binary_logloss: 0.481931\tvalid_0's auc: 0.768695\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[10]\tvalid_0's binary_logloss: 0.466103\tvalid_0's auc: 0.769675\n",
      "[11]\tvalid_0's binary_logloss: 0.460544\tvalid_0's auc: 0.769332\n",
      "[12]\tvalid_0's binary_logloss: 0.456187\tvalid_0's auc: 0.769133\n",
      "[13]\tvalid_0's binary_logloss: 0.452927\tvalid_0's auc: 0.76845\n",
      "[14]\tvalid_0's binary_logloss: 0.44965\tvalid_0's auc: 0.769096\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[1]\tvalid_0's binary_logloss: 0.640322\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.600099\tvalid_0's auc: 0.756076\n",
      "[3]\tvalid_0's binary_logloss: 0.567856\tvalid_0's auc: 0.761172\n",
      "[4]\tvalid_0's binary_logloss: 0.542555\tvalid_0's auc: 0.764108\n",
      "[5]\tvalid_0's binary_logloss: 0.522033\tvalid_0's auc: 0.765739\n",
      "[6]\tvalid_0's binary_logloss: 0.505883\tvalid_0's auc: 0.764746\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[8]\tvalid_0's binary_logloss: 0.482527\tvalid_0's auc: 0.763353\n",
      "[9]\tvalid_0's binary_logloss: 0.474211\tvalid_0's auc: 0.76342\n",
      "[10]\tvalid_0's binary_logloss: 0.467219\tvalid_0's auc: 0.763589\n",
      "[11]\tvalid_0's binary_logloss: 0.461432\tvalid_0's auc: 0.765676\n",
      "[12]\tvalid_0's binary_logloss: 0.456908\tvalid_0's auc: 0.766021\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[1]\tvalid_0's binary_logloss: 0.641006\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.60046\tvalid_0's auc: 0.751714\n",
      "[3]\tvalid_0's binary_logloss: 0.568982\tvalid_0's auc: 0.755611\n",
      "[4]\tvalid_0's binary_logloss: 0.543679\tvalid_0's auc: 0.75906\n",
      "[5]\tvalid_0's binary_logloss: 0.523356\tvalid_0's auc: 0.762148\n",
      "[6]\tvalid_0's binary_logloss: 0.507119\tvalid_0's auc: 0.760809\n",
      "[7]\tvalid_0's binary_logloss: 0.493783\tvalid_0's auc: 0.763779\n",
      "[8]\tvalid_0's binary_logloss: 0.482963\tvalid_0's auc: 0.764698\n",
      "[9]\tvalid_0's binary_logloss: 0.474584\tvalid_0's auc: 0.765261\n",
      "[10]\tvalid_0's binary_logloss: 0.467761\tvalid_0's auc: 0.76567\n",
      "[11]\tvalid_0's binary_logloss: 0.462211\tvalid_0's auc: 0.765548\n",
      "[12]\tvalid_0's binary_logloss: 0.457512\tvalid_0's auc: 0.766293\n",
      "[13]\tvalid_0's binary_logloss: 0.453961\tvalid_0's auc: 0.766222\n",
      "[14]\tvalid_0's binary_logloss: 0.451213\tvalid_0's auc: 0.766377\n",
      "[15]\tvalid_0's binary_logloss: 0.449126\tvalid_0's auc: 0.766314\n",
      "[16]\tvalid_0's binary_logloss: 0.447457\tvalid_0's auc: 0.765742\n",
      "[17]\tvalid_0's binary_logloss: 0.445718\tvalid_0's auc: 0.766506\n",
      "[18]\tvalid_0's binary_logloss: 0.444613\tvalid_0's auc: 0.766507\n",
      "[19]\tvalid_0's binary_logloss: 0.443095\tvalid_0's auc: 0.768662\n",
      "[20]\tvalid_0's binary_logloss: 0.442581\tvalid_0's auc: 0.768199\n",
      "[21]\tvalid_0's binary_logloss: 0.441855\tvalid_0's auc: 0.768631\n",
      "[22]\tvalid_0's binary_logloss: 0.441239\tvalid_0's auc: 0.769082\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[24]\tvalid_0's binary_logloss: 0.441121\tvalid_0's auc: 0.768284\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[26]\tvalid_0's binary_logloss: 0.440982\tvalid_0's auc: 0.767638\n",
      "[27]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.766989\n",
      "[28]\tvalid_0's binary_logloss: 0.441096\tvalid_0's auc: 0.766995\n",
      "Early stopping, best iteration is:\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[1]\tvalid_0's binary_logloss: 0.640346\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.599816\tvalid_0's auc: 0.756783\n",
      "[3]\tvalid_0's binary_logloss: 0.567876\tvalid_0's auc: 0.762467\n",
      "[4]\tvalid_0's binary_logloss: 0.542429\tvalid_0's auc: 0.763728\n",
      "[5]\tvalid_0's binary_logloss: 0.522117\tvalid_0's auc: 0.765465\n",
      "[6]\tvalid_0's binary_logloss: 0.506238\tvalid_0's auc: 0.765642\n",
      "[7]\tvalid_0's binary_logloss: 0.493102\tvalid_0's auc: 0.765213\n",
      "[8]\tvalid_0's binary_logloss: 0.481931\tvalid_0's auc: 0.768695\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[10]\tvalid_0's binary_logloss: 0.466103\tvalid_0's auc: 0.769675\n",
      "[11]\tvalid_0's binary_logloss: 0.460544\tvalid_0's auc: 0.769332\n",
      "[12]\tvalid_0's binary_logloss: 0.456187\tvalid_0's auc: 0.769133\n",
      "[13]\tvalid_0's binary_logloss: 0.452927\tvalid_0's auc: 0.76845\n",
      "[14]\tvalid_0's binary_logloss: 0.44965\tvalid_0's auc: 0.769096\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.640322\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.600099\tvalid_0's auc: 0.756076\n",
      "[3]\tvalid_0's binary_logloss: 0.567856\tvalid_0's auc: 0.761172\n",
      "[4]\tvalid_0's binary_logloss: 0.542555\tvalid_0's auc: 0.764108\n",
      "[5]\tvalid_0's binary_logloss: 0.522033\tvalid_0's auc: 0.765739\n",
      "[6]\tvalid_0's binary_logloss: 0.505883\tvalid_0's auc: 0.764746\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[8]\tvalid_0's binary_logloss: 0.482527\tvalid_0's auc: 0.763353\n",
      "[9]\tvalid_0's binary_logloss: 0.474211\tvalid_0's auc: 0.76342\n",
      "[10]\tvalid_0's binary_logloss: 0.467219\tvalid_0's auc: 0.763589\n",
      "[11]\tvalid_0's binary_logloss: 0.461432\tvalid_0's auc: 0.765676\n",
      "[12]\tvalid_0's binary_logloss: 0.456908\tvalid_0's auc: 0.766021\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[1]\tvalid_0's binary_logloss: 0.641006\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.60046\tvalid_0's auc: 0.751714\n",
      "[3]\tvalid_0's binary_logloss: 0.568982\tvalid_0's auc: 0.755611\n",
      "[4]\tvalid_0's binary_logloss: 0.543679\tvalid_0's auc: 0.75906\n",
      "[5]\tvalid_0's binary_logloss: 0.523356\tvalid_0's auc: 0.762148\n",
      "[6]\tvalid_0's binary_logloss: 0.507119\tvalid_0's auc: 0.760809\n",
      "[7]\tvalid_0's binary_logloss: 0.493783\tvalid_0's auc: 0.763779\n",
      "[8]\tvalid_0's binary_logloss: 0.482963\tvalid_0's auc: 0.764698\n",
      "[9]\tvalid_0's binary_logloss: 0.474584\tvalid_0's auc: 0.765261\n",
      "[10]\tvalid_0's binary_logloss: 0.467761\tvalid_0's auc: 0.76567\n",
      "[11]\tvalid_0's binary_logloss: 0.462211\tvalid_0's auc: 0.765548\n",
      "[12]\tvalid_0's binary_logloss: 0.457512\tvalid_0's auc: 0.766293\n",
      "[13]\tvalid_0's binary_logloss: 0.453961\tvalid_0's auc: 0.766222\n",
      "[14]\tvalid_0's binary_logloss: 0.451213\tvalid_0's auc: 0.766377\n",
      "[15]\tvalid_0's binary_logloss: 0.449126\tvalid_0's auc: 0.766314\n",
      "[16]\tvalid_0's binary_logloss: 0.447457\tvalid_0's auc: 0.765742\n",
      "[17]\tvalid_0's binary_logloss: 0.445718\tvalid_0's auc: 0.766506\n",
      "[18]\tvalid_0's binary_logloss: 0.444613\tvalid_0's auc: 0.766507\n",
      "[19]\tvalid_0's binary_logloss: 0.443095\tvalid_0's auc: 0.768662\n",
      "[20]\tvalid_0's binary_logloss: 0.442581\tvalid_0's auc: 0.768199\n",
      "[21]\tvalid_0's binary_logloss: 0.441855\tvalid_0's auc: 0.768631\n",
      "[22]\tvalid_0's binary_logloss: 0.441239\tvalid_0's auc: 0.769082\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[24]\tvalid_0's binary_logloss: 0.441121\tvalid_0's auc: 0.768284\n",
      "[25]\tvalid_0's binary_logloss: 0.44085\tvalid_0's auc: 0.767877\n",
      "[26]\tvalid_0's binary_logloss: 0.440982\tvalid_0's auc: 0.767638\n",
      "[27]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.766989\n",
      "[28]\tvalid_0's binary_logloss: 0.441096\tvalid_0's auc: 0.766995\n",
      "Early stopping, best iteration is:\n",
      "[23]\tvalid_0's binary_logloss: 0.440852\tvalid_0's auc: 0.769305\n",
      "[1]\tvalid_0's binary_logloss: 0.640346\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.599816\tvalid_0's auc: 0.756783\n",
      "[3]\tvalid_0's binary_logloss: 0.567876\tvalid_0's auc: 0.762467\n",
      "[4]\tvalid_0's binary_logloss: 0.542429\tvalid_0's auc: 0.763728\n",
      "[5]\tvalid_0's binary_logloss: 0.522117\tvalid_0's auc: 0.765465\n",
      "[6]\tvalid_0's binary_logloss: 0.506238\tvalid_0's auc: 0.765642\n",
      "[7]\tvalid_0's binary_logloss: 0.493102\tvalid_0's auc: 0.765213\n",
      "[8]\tvalid_0's binary_logloss: 0.481931\tvalid_0's auc: 0.768695\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[10]\tvalid_0's binary_logloss: 0.466103\tvalid_0's auc: 0.769675\n",
      "[11]\tvalid_0's binary_logloss: 0.460544\tvalid_0's auc: 0.769332\n",
      "[12]\tvalid_0's binary_logloss: 0.456187\tvalid_0's auc: 0.769133\n",
      "[13]\tvalid_0's binary_logloss: 0.452927\tvalid_0's auc: 0.76845\n",
      "[14]\tvalid_0's binary_logloss: 0.44965\tvalid_0's auc: 0.769096\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.473262\tvalid_0's auc: 0.769823\n",
      "[1]\tvalid_0's binary_logloss: 0.640322\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.600099\tvalid_0's auc: 0.756076\n",
      "[3]\tvalid_0's binary_logloss: 0.567856\tvalid_0's auc: 0.761172\n",
      "[4]\tvalid_0's binary_logloss: 0.542555\tvalid_0's auc: 0.764108\n",
      "[5]\tvalid_0's binary_logloss: 0.522033\tvalid_0's auc: 0.765739\n",
      "[6]\tvalid_0's binary_logloss: 0.505883\tvalid_0's auc: 0.764746\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[8]\tvalid_0's binary_logloss: 0.482527\tvalid_0's auc: 0.763353\n",
      "[9]\tvalid_0's binary_logloss: 0.474211\tvalid_0's auc: 0.76342\n",
      "[10]\tvalid_0's binary_logloss: 0.467219\tvalid_0's auc: 0.763589\n",
      "[11]\tvalid_0's binary_logloss: 0.461432\tvalid_0's auc: 0.765676\n",
      "[12]\tvalid_0's binary_logloss: 0.456908\tvalid_0's auc: 0.766021\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.492622\tvalid_0's auc: 0.766029\n",
      "[1]\tvalid_0's binary_logloss: 0.631448\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.586118\tvalid_0's auc: 0.751527\n",
      "[3]\tvalid_0's binary_logloss: 0.552221\tvalid_0's auc: 0.756667\n",
      "[4]\tvalid_0's binary_logloss: 0.526307\tvalid_0's auc: 0.759955\n",
      "[5]\tvalid_0's binary_logloss: 0.506382\tvalid_0's auc: 0.760804\n",
      "[6]\tvalid_0's binary_logloss: 0.490891\tvalid_0's auc: 0.763243\n",
      "[7]\tvalid_0's binary_logloss: 0.478964\tvalid_0's auc: 0.764198\n",
      "[8]\tvalid_0's binary_logloss: 0.46972\tvalid_0's auc: 0.764964\n",
      "[9]\tvalid_0's binary_logloss: 0.462649\tvalid_0's auc: 0.76587\n",
      "[10]\tvalid_0's binary_logloss: 0.457395\tvalid_0's auc: 0.765755\n",
      "[11]\tvalid_0's binary_logloss: 0.453074\tvalid_0's auc: 0.766078\n",
      "[12]\tvalid_0's binary_logloss: 0.449682\tvalid_0's auc: 0.766941\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[14]\tvalid_0's binary_logloss: 0.445637\tvalid_0's auc: 0.76661\n",
      "[15]\tvalid_0's binary_logloss: 0.444598\tvalid_0's auc: 0.765829\n",
      "[16]\tvalid_0's binary_logloss: 0.443679\tvalid_0's auc: 0.765888\n",
      "[17]\tvalid_0's binary_logloss: 0.442791\tvalid_0's auc: 0.766334\n",
      "[18]\tvalid_0's binary_logloss: 0.442223\tvalid_0's auc: 0.76699\n",
      "Early stopping, best iteration is:\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[1]\tvalid_0's binary_logloss: 0.630654\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585205\tvalid_0's auc: 0.757088\n",
      "[3]\tvalid_0's binary_logloss: 0.550895\tvalid_0's auc: 0.761995\n",
      "[4]\tvalid_0's binary_logloss: 0.524251\tvalid_0's auc: 0.766984\n",
      "[5]\tvalid_0's binary_logloss: 0.504433\tvalid_0's auc: 0.770568\n",
      "[6]\tvalid_0's binary_logloss: 0.489087\tvalid_0's auc: 0.770545\n",
      "[7]\tvalid_0's binary_logloss: 0.47757\tvalid_0's auc: 0.770446\n",
      "[8]\tvalid_0's binary_logloss: 0.468269\tvalid_0's auc: 0.770626\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[10]\tvalid_0's binary_logloss: 0.455769\tvalid_0's auc: 0.769739\n",
      "[11]\tvalid_0's binary_logloss: 0.451533\tvalid_0's auc: 0.769676\n",
      "[12]\tvalid_0's binary_logloss: 0.448643\tvalid_0's auc: 0.769198\n",
      "[13]\tvalid_0's binary_logloss: 0.446163\tvalid_0's auc: 0.77009\n",
      "[14]\tvalid_0's binary_logloss: 0.444737\tvalid_0's auc: 0.769141\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[1]\tvalid_0's binary_logloss: 0.630622\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585383\tvalid_0's auc: 0.755866\n",
      "[3]\tvalid_0's binary_logloss: 0.550719\tvalid_0's auc: 0.762993\n",
      "[4]\tvalid_0's binary_logloss: 0.524539\tvalid_0's auc: 0.764337\n",
      "[5]\tvalid_0's binary_logloss: 0.504647\tvalid_0's auc: 0.765675\n",
      "[6]\tvalid_0's binary_logloss: 0.489179\tvalid_0's auc: 0.76446\n",
      "[7]\tvalid_0's binary_logloss: 0.47741\tvalid_0's auc: 0.765702\n",
      "[8]\tvalid_0's binary_logloss: 0.468487\tvalid_0's auc: 0.765607\n",
      "[9]\tvalid_0's binary_logloss: 0.461515\tvalid_0's auc: 0.766937\n",
      "[10]\tvalid_0's binary_logloss: 0.455857\tvalid_0's auc: 0.767589\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[12]\tvalid_0's binary_logloss: 0.448367\tvalid_0's auc: 0.768973\n",
      "[13]\tvalid_0's binary_logloss: 0.446381\tvalid_0's auc: 0.768027\n",
      "[14]\tvalid_0's binary_logloss: 0.444324\tvalid_0's auc: 0.768433\n",
      "[15]\tvalid_0's binary_logloss: 0.443134\tvalid_0's auc: 0.768511\n",
      "[16]\tvalid_0's binary_logloss: 0.442375\tvalid_0's auc: 0.767896\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.631448\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.586118\tvalid_0's auc: 0.751527\n",
      "[3]\tvalid_0's binary_logloss: 0.552221\tvalid_0's auc: 0.756667\n",
      "[4]\tvalid_0's binary_logloss: 0.526307\tvalid_0's auc: 0.759955\n",
      "[5]\tvalid_0's binary_logloss: 0.506382\tvalid_0's auc: 0.760804\n",
      "[6]\tvalid_0's binary_logloss: 0.490891\tvalid_0's auc: 0.763243\n",
      "[7]\tvalid_0's binary_logloss: 0.478964\tvalid_0's auc: 0.764198\n",
      "[8]\tvalid_0's binary_logloss: 0.46972\tvalid_0's auc: 0.764964\n",
      "[9]\tvalid_0's binary_logloss: 0.462649\tvalid_0's auc: 0.76587\n",
      "[10]\tvalid_0's binary_logloss: 0.457395\tvalid_0's auc: 0.765755\n",
      "[11]\tvalid_0's binary_logloss: 0.453074\tvalid_0's auc: 0.766078\n",
      "[12]\tvalid_0's binary_logloss: 0.449682\tvalid_0's auc: 0.766941\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[14]\tvalid_0's binary_logloss: 0.445637\tvalid_0's auc: 0.76661\n",
      "[15]\tvalid_0's binary_logloss: 0.444598\tvalid_0's auc: 0.765829\n",
      "[16]\tvalid_0's binary_logloss: 0.443679\tvalid_0's auc: 0.765888\n",
      "[17]\tvalid_0's binary_logloss: 0.442791\tvalid_0's auc: 0.766334\n",
      "[18]\tvalid_0's binary_logloss: 0.442223\tvalid_0's auc: 0.76699\n",
      "Early stopping, best iteration is:\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[1]\tvalid_0's binary_logloss: 0.630654\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585205\tvalid_0's auc: 0.757088\n",
      "[3]\tvalid_0's binary_logloss: 0.550895\tvalid_0's auc: 0.761995\n",
      "[4]\tvalid_0's binary_logloss: 0.524251\tvalid_0's auc: 0.766984\n",
      "[5]\tvalid_0's binary_logloss: 0.504433\tvalid_0's auc: 0.770568\n",
      "[6]\tvalid_0's binary_logloss: 0.489087\tvalid_0's auc: 0.770545\n",
      "[7]\tvalid_0's binary_logloss: 0.47757\tvalid_0's auc: 0.770446\n",
      "[8]\tvalid_0's binary_logloss: 0.468269\tvalid_0's auc: 0.770626\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[10]\tvalid_0's binary_logloss: 0.455769\tvalid_0's auc: 0.769739\n",
      "[11]\tvalid_0's binary_logloss: 0.451533\tvalid_0's auc: 0.769676\n",
      "[12]\tvalid_0's binary_logloss: 0.448643\tvalid_0's auc: 0.769198\n",
      "[13]\tvalid_0's binary_logloss: 0.446163\tvalid_0's auc: 0.77009\n",
      "[14]\tvalid_0's binary_logloss: 0.444737\tvalid_0's auc: 0.769141\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[1]\tvalid_0's binary_logloss: 0.630622\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585383\tvalid_0's auc: 0.755866\n",
      "[3]\tvalid_0's binary_logloss: 0.550719\tvalid_0's auc: 0.762993\n",
      "[4]\tvalid_0's binary_logloss: 0.524539\tvalid_0's auc: 0.764337\n",
      "[5]\tvalid_0's binary_logloss: 0.504647\tvalid_0's auc: 0.765675\n",
      "[6]\tvalid_0's binary_logloss: 0.489179\tvalid_0's auc: 0.76446\n",
      "[7]\tvalid_0's binary_logloss: 0.47741\tvalid_0's auc: 0.765702\n",
      "[8]\tvalid_0's binary_logloss: 0.468487\tvalid_0's auc: 0.765607\n",
      "[9]\tvalid_0's binary_logloss: 0.461515\tvalid_0's auc: 0.766937\n",
      "[10]\tvalid_0's binary_logloss: 0.455857\tvalid_0's auc: 0.767589\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[12]\tvalid_0's binary_logloss: 0.448367\tvalid_0's auc: 0.768973\n",
      "[13]\tvalid_0's binary_logloss: 0.446381\tvalid_0's auc: 0.768027\n",
      "[14]\tvalid_0's binary_logloss: 0.444324\tvalid_0's auc: 0.768433\n",
      "[15]\tvalid_0's binary_logloss: 0.443134\tvalid_0's auc: 0.768511\n",
      "[16]\tvalid_0's binary_logloss: 0.442375\tvalid_0's auc: 0.767896\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[1]\tvalid_0's binary_logloss: 0.631448\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.586118\tvalid_0's auc: 0.751527\n",
      "[3]\tvalid_0's binary_logloss: 0.552221\tvalid_0's auc: 0.756667\n",
      "[4]\tvalid_0's binary_logloss: 0.526307\tvalid_0's auc: 0.759955\n",
      "[5]\tvalid_0's binary_logloss: 0.506382\tvalid_0's auc: 0.760804\n",
      "[6]\tvalid_0's binary_logloss: 0.490891\tvalid_0's auc: 0.763243\n",
      "[7]\tvalid_0's binary_logloss: 0.478964\tvalid_0's auc: 0.764198\n",
      "[8]\tvalid_0's binary_logloss: 0.46972\tvalid_0's auc: 0.764964\n",
      "[9]\tvalid_0's binary_logloss: 0.462649\tvalid_0's auc: 0.76587\n",
      "[10]\tvalid_0's binary_logloss: 0.457395\tvalid_0's auc: 0.765755\n",
      "[11]\tvalid_0's binary_logloss: 0.453074\tvalid_0's auc: 0.766078\n",
      "[12]\tvalid_0's binary_logloss: 0.449682\tvalid_0's auc: 0.766941\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[14]\tvalid_0's binary_logloss: 0.445637\tvalid_0's auc: 0.76661\n",
      "[15]\tvalid_0's binary_logloss: 0.444598\tvalid_0's auc: 0.765829\n",
      "[16]\tvalid_0's binary_logloss: 0.443679\tvalid_0's auc: 0.765888\n",
      "[17]\tvalid_0's binary_logloss: 0.442791\tvalid_0's auc: 0.766334\n",
      "[18]\tvalid_0's binary_logloss: 0.442223\tvalid_0's auc: 0.76699\n",
      "Early stopping, best iteration is:\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[1]\tvalid_0's binary_logloss: 0.630654\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585205\tvalid_0's auc: 0.757088\n",
      "[3]\tvalid_0's binary_logloss: 0.550895\tvalid_0's auc: 0.761995\n",
      "[4]\tvalid_0's binary_logloss: 0.524251\tvalid_0's auc: 0.766984\n",
      "[5]\tvalid_0's binary_logloss: 0.504433\tvalid_0's auc: 0.770568\n",
      "[6]\tvalid_0's binary_logloss: 0.489087\tvalid_0's auc: 0.770545\n",
      "[7]\tvalid_0's binary_logloss: 0.47757\tvalid_0's auc: 0.770446\n",
      "[8]\tvalid_0's binary_logloss: 0.468269\tvalid_0's auc: 0.770626\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[10]\tvalid_0's binary_logloss: 0.455769\tvalid_0's auc: 0.769739\n",
      "[11]\tvalid_0's binary_logloss: 0.451533\tvalid_0's auc: 0.769676\n",
      "[12]\tvalid_0's binary_logloss: 0.448643\tvalid_0's auc: 0.769198\n",
      "[13]\tvalid_0's binary_logloss: 0.446163\tvalid_0's auc: 0.77009\n",
      "[14]\tvalid_0's binary_logloss: 0.444737\tvalid_0's auc: 0.769141\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[1]\tvalid_0's binary_logloss: 0.630622\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585383\tvalid_0's auc: 0.755866\n",
      "[3]\tvalid_0's binary_logloss: 0.550719\tvalid_0's auc: 0.762993\n",
      "[4]\tvalid_0's binary_logloss: 0.524539\tvalid_0's auc: 0.764337\n",
      "[5]\tvalid_0's binary_logloss: 0.504647\tvalid_0's auc: 0.765675\n",
      "[6]\tvalid_0's binary_logloss: 0.489179\tvalid_0's auc: 0.76446\n",
      "[7]\tvalid_0's binary_logloss: 0.47741\tvalid_0's auc: 0.765702\n",
      "[8]\tvalid_0's binary_logloss: 0.468487\tvalid_0's auc: 0.765607\n",
      "[9]\tvalid_0's binary_logloss: 0.461515\tvalid_0's auc: 0.766937\n",
      "[10]\tvalid_0's binary_logloss: 0.455857\tvalid_0's auc: 0.767589\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[12]\tvalid_0's binary_logloss: 0.448367\tvalid_0's auc: 0.768973\n",
      "[13]\tvalid_0's binary_logloss: 0.446381\tvalid_0's auc: 0.768027\n",
      "[14]\tvalid_0's binary_logloss: 0.444324\tvalid_0's auc: 0.768433\n",
      "[15]\tvalid_0's binary_logloss: 0.443134\tvalid_0's auc: 0.768511\n",
      "[16]\tvalid_0's binary_logloss: 0.442375\tvalid_0's auc: 0.767896\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[1]\tvalid_0's binary_logloss: 0.631448\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.586118\tvalid_0's auc: 0.751527\n",
      "[3]\tvalid_0's binary_logloss: 0.552221\tvalid_0's auc: 0.756667\n",
      "[4]\tvalid_0's binary_logloss: 0.526307\tvalid_0's auc: 0.759955\n",
      "[5]\tvalid_0's binary_logloss: 0.506382\tvalid_0's auc: 0.760804\n",
      "[6]\tvalid_0's binary_logloss: 0.490891\tvalid_0's auc: 0.763243\n",
      "[7]\tvalid_0's binary_logloss: 0.478964\tvalid_0's auc: 0.764198\n",
      "[8]\tvalid_0's binary_logloss: 0.46972\tvalid_0's auc: 0.764964\n",
      "[9]\tvalid_0's binary_logloss: 0.462649\tvalid_0's auc: 0.76587\n",
      "[10]\tvalid_0's binary_logloss: 0.457395\tvalid_0's auc: 0.765755\n",
      "[11]\tvalid_0's binary_logloss: 0.453074\tvalid_0's auc: 0.766078\n",
      "[12]\tvalid_0's binary_logloss: 0.449682\tvalid_0's auc: 0.766941\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[14]\tvalid_0's binary_logloss: 0.445637\tvalid_0's auc: 0.76661\n",
      "[15]\tvalid_0's binary_logloss: 0.444598\tvalid_0's auc: 0.765829\n",
      "[16]\tvalid_0's binary_logloss: 0.443679\tvalid_0's auc: 0.765888\n",
      "[17]\tvalid_0's binary_logloss: 0.442791\tvalid_0's auc: 0.766334\n",
      "[18]\tvalid_0's binary_logloss: 0.442223\tvalid_0's auc: 0.76699\n",
      "Early stopping, best iteration is:\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[1]\tvalid_0's binary_logloss: 0.630654\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585205\tvalid_0's auc: 0.757088\n",
      "[3]\tvalid_0's binary_logloss: 0.550895\tvalid_0's auc: 0.761995\n",
      "[4]\tvalid_0's binary_logloss: 0.524251\tvalid_0's auc: 0.766984\n",
      "[5]\tvalid_0's binary_logloss: 0.504433\tvalid_0's auc: 0.770568\n",
      "[6]\tvalid_0's binary_logloss: 0.489087\tvalid_0's auc: 0.770545\n",
      "[7]\tvalid_0's binary_logloss: 0.47757\tvalid_0's auc: 0.770446\n",
      "[8]\tvalid_0's binary_logloss: 0.468269\tvalid_0's auc: 0.770626\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[10]\tvalid_0's binary_logloss: 0.455769\tvalid_0's auc: 0.769739\n",
      "[11]\tvalid_0's binary_logloss: 0.451533\tvalid_0's auc: 0.769676\n",
      "[12]\tvalid_0's binary_logloss: 0.448643\tvalid_0's auc: 0.769198\n",
      "[13]\tvalid_0's binary_logloss: 0.446163\tvalid_0's auc: 0.77009\n",
      "[14]\tvalid_0's binary_logloss: 0.444737\tvalid_0's auc: 0.769141\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.630622\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585383\tvalid_0's auc: 0.755866\n",
      "[3]\tvalid_0's binary_logloss: 0.550719\tvalid_0's auc: 0.762993\n",
      "[4]\tvalid_0's binary_logloss: 0.524539\tvalid_0's auc: 0.764337\n",
      "[5]\tvalid_0's binary_logloss: 0.504647\tvalid_0's auc: 0.765675\n",
      "[6]\tvalid_0's binary_logloss: 0.489179\tvalid_0's auc: 0.76446\n",
      "[7]\tvalid_0's binary_logloss: 0.47741\tvalid_0's auc: 0.765702\n",
      "[8]\tvalid_0's binary_logloss: 0.468487\tvalid_0's auc: 0.765607\n",
      "[9]\tvalid_0's binary_logloss: 0.461515\tvalid_0's auc: 0.766937\n",
      "[10]\tvalid_0's binary_logloss: 0.455857\tvalid_0's auc: 0.767589\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[12]\tvalid_0's binary_logloss: 0.448367\tvalid_0's auc: 0.768973\n",
      "[13]\tvalid_0's binary_logloss: 0.446381\tvalid_0's auc: 0.768027\n",
      "[14]\tvalid_0's binary_logloss: 0.444324\tvalid_0's auc: 0.768433\n",
      "[15]\tvalid_0's binary_logloss: 0.443134\tvalid_0's auc: 0.768511\n",
      "[16]\tvalid_0's binary_logloss: 0.442375\tvalid_0's auc: 0.767896\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[1]\tvalid_0's binary_logloss: 0.631448\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.586118\tvalid_0's auc: 0.751527\n",
      "[3]\tvalid_0's binary_logloss: 0.552221\tvalid_0's auc: 0.756667\n",
      "[4]\tvalid_0's binary_logloss: 0.526307\tvalid_0's auc: 0.759955\n",
      "[5]\tvalid_0's binary_logloss: 0.506382\tvalid_0's auc: 0.760804\n",
      "[6]\tvalid_0's binary_logloss: 0.490891\tvalid_0's auc: 0.763243\n",
      "[7]\tvalid_0's binary_logloss: 0.478964\tvalid_0's auc: 0.764198\n",
      "[8]\tvalid_0's binary_logloss: 0.46972\tvalid_0's auc: 0.764964\n",
      "[9]\tvalid_0's binary_logloss: 0.462649\tvalid_0's auc: 0.76587\n",
      "[10]\tvalid_0's binary_logloss: 0.457395\tvalid_0's auc: 0.765755\n",
      "[11]\tvalid_0's binary_logloss: 0.453074\tvalid_0's auc: 0.766078\n",
      "[12]\tvalid_0's binary_logloss: 0.449682\tvalid_0's auc: 0.766941\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[14]\tvalid_0's binary_logloss: 0.445637\tvalid_0's auc: 0.76661\n",
      "[15]\tvalid_0's binary_logloss: 0.444598\tvalid_0's auc: 0.765829\n",
      "[16]\tvalid_0's binary_logloss: 0.443679\tvalid_0's auc: 0.765888\n",
      "[17]\tvalid_0's binary_logloss: 0.442791\tvalid_0's auc: 0.766334\n",
      "[18]\tvalid_0's binary_logloss: 0.442223\tvalid_0's auc: 0.76699\n",
      "Early stopping, best iteration is:\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[1]\tvalid_0's binary_logloss: 0.630654\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585205\tvalid_0's auc: 0.757088\n",
      "[3]\tvalid_0's binary_logloss: 0.550895\tvalid_0's auc: 0.761995\n",
      "[4]\tvalid_0's binary_logloss: 0.524251\tvalid_0's auc: 0.766984\n",
      "[5]\tvalid_0's binary_logloss: 0.504433\tvalid_0's auc: 0.770568\n",
      "[6]\tvalid_0's binary_logloss: 0.489087\tvalid_0's auc: 0.770545\n",
      "[7]\tvalid_0's binary_logloss: 0.47757\tvalid_0's auc: 0.770446\n",
      "[8]\tvalid_0's binary_logloss: 0.468269\tvalid_0's auc: 0.770626\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[10]\tvalid_0's binary_logloss: 0.455769\tvalid_0's auc: 0.769739\n",
      "[11]\tvalid_0's binary_logloss: 0.451533\tvalid_0's auc: 0.769676\n",
      "[12]\tvalid_0's binary_logloss: 0.448643\tvalid_0's auc: 0.769198\n",
      "[13]\tvalid_0's binary_logloss: 0.446163\tvalid_0's auc: 0.77009\n",
      "[14]\tvalid_0's binary_logloss: 0.444737\tvalid_0's auc: 0.769141\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[1]\tvalid_0's binary_logloss: 0.630622\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585383\tvalid_0's auc: 0.755866\n",
      "[3]\tvalid_0's binary_logloss: 0.550719\tvalid_0's auc: 0.762993\n",
      "[4]\tvalid_0's binary_logloss: 0.524539\tvalid_0's auc: 0.764337\n",
      "[5]\tvalid_0's binary_logloss: 0.504647\tvalid_0's auc: 0.765675\n",
      "[6]\tvalid_0's binary_logloss: 0.489179\tvalid_0's auc: 0.76446\n",
      "[7]\tvalid_0's binary_logloss: 0.47741\tvalid_0's auc: 0.765702\n",
      "[8]\tvalid_0's binary_logloss: 0.468487\tvalid_0's auc: 0.765607\n",
      "[9]\tvalid_0's binary_logloss: 0.461515\tvalid_0's auc: 0.766937\n",
      "[10]\tvalid_0's binary_logloss: 0.455857\tvalid_0's auc: 0.767589\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[12]\tvalid_0's binary_logloss: 0.448367\tvalid_0's auc: 0.768973\n",
      "[13]\tvalid_0's binary_logloss: 0.446381\tvalid_0's auc: 0.768027\n",
      "[14]\tvalid_0's binary_logloss: 0.444324\tvalid_0's auc: 0.768433\n",
      "[15]\tvalid_0's binary_logloss: 0.443134\tvalid_0's auc: 0.768511\n",
      "[16]\tvalid_0's binary_logloss: 0.442375\tvalid_0's auc: 0.767896\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[1]\tvalid_0's binary_logloss: 0.631448\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.586118\tvalid_0's auc: 0.751527\n",
      "[3]\tvalid_0's binary_logloss: 0.552221\tvalid_0's auc: 0.756667\n",
      "[4]\tvalid_0's binary_logloss: 0.526307\tvalid_0's auc: 0.759955\n",
      "[5]\tvalid_0's binary_logloss: 0.506382\tvalid_0's auc: 0.760804\n",
      "[6]\tvalid_0's binary_logloss: 0.490891\tvalid_0's auc: 0.763243\n",
      "[7]\tvalid_0's binary_logloss: 0.478964\tvalid_0's auc: 0.764198\n",
      "[8]\tvalid_0's binary_logloss: 0.46972\tvalid_0's auc: 0.764964\n",
      "[9]\tvalid_0's binary_logloss: 0.462649\tvalid_0's auc: 0.76587\n",
      "[10]\tvalid_0's binary_logloss: 0.457395\tvalid_0's auc: 0.765755\n",
      "[11]\tvalid_0's binary_logloss: 0.453074\tvalid_0's auc: 0.766078\n",
      "[12]\tvalid_0's binary_logloss: 0.449682\tvalid_0's auc: 0.766941\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[14]\tvalid_0's binary_logloss: 0.445637\tvalid_0's auc: 0.76661\n",
      "[15]\tvalid_0's binary_logloss: 0.444598\tvalid_0's auc: 0.765829\n",
      "[16]\tvalid_0's binary_logloss: 0.443679\tvalid_0's auc: 0.765888\n",
      "[17]\tvalid_0's binary_logloss: 0.442791\tvalid_0's auc: 0.766334\n",
      "[18]\tvalid_0's binary_logloss: 0.442223\tvalid_0's auc: 0.76699\n",
      "Early stopping, best iteration is:\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[1]\tvalid_0's binary_logloss: 0.630654\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585205\tvalid_0's auc: 0.757088\n",
      "[3]\tvalid_0's binary_logloss: 0.550895\tvalid_0's auc: 0.761995\n",
      "[4]\tvalid_0's binary_logloss: 0.524251\tvalid_0's auc: 0.766984\n",
      "[5]\tvalid_0's binary_logloss: 0.504433\tvalid_0's auc: 0.770568\n",
      "[6]\tvalid_0's binary_logloss: 0.489087\tvalid_0's auc: 0.770545\n",
      "[7]\tvalid_0's binary_logloss: 0.47757\tvalid_0's auc: 0.770446\n",
      "[8]\tvalid_0's binary_logloss: 0.468269\tvalid_0's auc: 0.770626\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[10]\tvalid_0's binary_logloss: 0.455769\tvalid_0's auc: 0.769739\n",
      "[11]\tvalid_0's binary_logloss: 0.451533\tvalid_0's auc: 0.769676\n",
      "[12]\tvalid_0's binary_logloss: 0.448643\tvalid_0's auc: 0.769198\n",
      "[13]\tvalid_0's binary_logloss: 0.446163\tvalid_0's auc: 0.77009\n",
      "[14]\tvalid_0's binary_logloss: 0.444737\tvalid_0's auc: 0.769141\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[1]\tvalid_0's binary_logloss: 0.630622\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585383\tvalid_0's auc: 0.755866\n",
      "[3]\tvalid_0's binary_logloss: 0.550719\tvalid_0's auc: 0.762993\n",
      "[4]\tvalid_0's binary_logloss: 0.524539\tvalid_0's auc: 0.764337\n",
      "[5]\tvalid_0's binary_logloss: 0.504647\tvalid_0's auc: 0.765675\n",
      "[6]\tvalid_0's binary_logloss: 0.489179\tvalid_0's auc: 0.76446\n",
      "[7]\tvalid_0's binary_logloss: 0.47741\tvalid_0's auc: 0.765702\n",
      "[8]\tvalid_0's binary_logloss: 0.468487\tvalid_0's auc: 0.765607\n",
      "[9]\tvalid_0's binary_logloss: 0.461515\tvalid_0's auc: 0.766937\n",
      "[10]\tvalid_0's binary_logloss: 0.455857\tvalid_0's auc: 0.767589\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[12]\tvalid_0's binary_logloss: 0.448367\tvalid_0's auc: 0.768973\n",
      "[13]\tvalid_0's binary_logloss: 0.446381\tvalid_0's auc: 0.768027\n",
      "[14]\tvalid_0's binary_logloss: 0.444324\tvalid_0's auc: 0.768433\n",
      "[15]\tvalid_0's binary_logloss: 0.443134\tvalid_0's auc: 0.768511\n",
      "[16]\tvalid_0's binary_logloss: 0.442375\tvalid_0's auc: 0.767896\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[1]\tvalid_0's binary_logloss: 0.631448\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.586118\tvalid_0's auc: 0.751527\n",
      "[3]\tvalid_0's binary_logloss: 0.552221\tvalid_0's auc: 0.756667\n",
      "[4]\tvalid_0's binary_logloss: 0.526307\tvalid_0's auc: 0.759955\n",
      "[5]\tvalid_0's binary_logloss: 0.506382\tvalid_0's auc: 0.760804\n",
      "[6]\tvalid_0's binary_logloss: 0.490891\tvalid_0's auc: 0.763243\n",
      "[7]\tvalid_0's binary_logloss: 0.478964\tvalid_0's auc: 0.764198\n",
      "[8]\tvalid_0's binary_logloss: 0.46972\tvalid_0's auc: 0.764964\n",
      "[9]\tvalid_0's binary_logloss: 0.462649\tvalid_0's auc: 0.76587\n",
      "[10]\tvalid_0's binary_logloss: 0.457395\tvalid_0's auc: 0.765755\n",
      "[11]\tvalid_0's binary_logloss: 0.453074\tvalid_0's auc: 0.766078\n",
      "[12]\tvalid_0's binary_logloss: 0.449682\tvalid_0's auc: 0.766941\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[14]\tvalid_0's binary_logloss: 0.445637\tvalid_0's auc: 0.76661\n",
      "[15]\tvalid_0's binary_logloss: 0.444598\tvalid_0's auc: 0.765829\n",
      "[16]\tvalid_0's binary_logloss: 0.443679\tvalid_0's auc: 0.765888\n",
      "[17]\tvalid_0's binary_logloss: 0.442791\tvalid_0's auc: 0.766334\n",
      "[18]\tvalid_0's binary_logloss: 0.442223\tvalid_0's auc: 0.76699\n",
      "Early stopping, best iteration is:\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.630654\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585205\tvalid_0's auc: 0.757088\n",
      "[3]\tvalid_0's binary_logloss: 0.550895\tvalid_0's auc: 0.761995\n",
      "[4]\tvalid_0's binary_logloss: 0.524251\tvalid_0's auc: 0.766984\n",
      "[5]\tvalid_0's binary_logloss: 0.504433\tvalid_0's auc: 0.770568\n",
      "[6]\tvalid_0's binary_logloss: 0.489087\tvalid_0's auc: 0.770545\n",
      "[7]\tvalid_0's binary_logloss: 0.47757\tvalid_0's auc: 0.770446\n",
      "[8]\tvalid_0's binary_logloss: 0.468269\tvalid_0's auc: 0.770626\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[10]\tvalid_0's binary_logloss: 0.455769\tvalid_0's auc: 0.769739\n",
      "[11]\tvalid_0's binary_logloss: 0.451533\tvalid_0's auc: 0.769676\n",
      "[12]\tvalid_0's binary_logloss: 0.448643\tvalid_0's auc: 0.769198\n",
      "[13]\tvalid_0's binary_logloss: 0.446163\tvalid_0's auc: 0.77009\n",
      "[14]\tvalid_0's binary_logloss: 0.444737\tvalid_0's auc: 0.769141\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[1]\tvalid_0's binary_logloss: 0.630622\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585383\tvalid_0's auc: 0.755866\n",
      "[3]\tvalid_0's binary_logloss: 0.550719\tvalid_0's auc: 0.762993\n",
      "[4]\tvalid_0's binary_logloss: 0.524539\tvalid_0's auc: 0.764337\n",
      "[5]\tvalid_0's binary_logloss: 0.504647\tvalid_0's auc: 0.765675\n",
      "[6]\tvalid_0's binary_logloss: 0.489179\tvalid_0's auc: 0.76446\n",
      "[7]\tvalid_0's binary_logloss: 0.47741\tvalid_0's auc: 0.765702\n",
      "[8]\tvalid_0's binary_logloss: 0.468487\tvalid_0's auc: 0.765607\n",
      "[9]\tvalid_0's binary_logloss: 0.461515\tvalid_0's auc: 0.766937\n",
      "[10]\tvalid_0's binary_logloss: 0.455857\tvalid_0's auc: 0.767589\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[12]\tvalid_0's binary_logloss: 0.448367\tvalid_0's auc: 0.768973\n",
      "[13]\tvalid_0's binary_logloss: 0.446381\tvalid_0's auc: 0.768027\n",
      "[14]\tvalid_0's binary_logloss: 0.444324\tvalid_0's auc: 0.768433\n",
      "[15]\tvalid_0's binary_logloss: 0.443134\tvalid_0's auc: 0.768511\n",
      "[16]\tvalid_0's binary_logloss: 0.442375\tvalid_0's auc: 0.767896\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[1]\tvalid_0's binary_logloss: 0.631448\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.586118\tvalid_0's auc: 0.751527\n",
      "[3]\tvalid_0's binary_logloss: 0.552221\tvalid_0's auc: 0.756667\n",
      "[4]\tvalid_0's binary_logloss: 0.526307\tvalid_0's auc: 0.759955\n",
      "[5]\tvalid_0's binary_logloss: 0.506382\tvalid_0's auc: 0.760804\n",
      "[6]\tvalid_0's binary_logloss: 0.490891\tvalid_0's auc: 0.763243\n",
      "[7]\tvalid_0's binary_logloss: 0.478964\tvalid_0's auc: 0.764198\n",
      "[8]\tvalid_0's binary_logloss: 0.46972\tvalid_0's auc: 0.764964\n",
      "[9]\tvalid_0's binary_logloss: 0.462649\tvalid_0's auc: 0.76587\n",
      "[10]\tvalid_0's binary_logloss: 0.457395\tvalid_0's auc: 0.765755\n",
      "[11]\tvalid_0's binary_logloss: 0.453074\tvalid_0's auc: 0.766078\n",
      "[12]\tvalid_0's binary_logloss: 0.449682\tvalid_0's auc: 0.766941\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[14]\tvalid_0's binary_logloss: 0.445637\tvalid_0's auc: 0.76661\n",
      "[15]\tvalid_0's binary_logloss: 0.444598\tvalid_0's auc: 0.765829\n",
      "[16]\tvalid_0's binary_logloss: 0.443679\tvalid_0's auc: 0.765888\n",
      "[17]\tvalid_0's binary_logloss: 0.442791\tvalid_0's auc: 0.766334\n",
      "[18]\tvalid_0's binary_logloss: 0.442223\tvalid_0's auc: 0.76699\n",
      "Early stopping, best iteration is:\n",
      "[13]\tvalid_0's binary_logloss: 0.447262\tvalid_0's auc: 0.76747\n",
      "[1]\tvalid_0's binary_logloss: 0.630654\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585205\tvalid_0's auc: 0.757088\n",
      "[3]\tvalid_0's binary_logloss: 0.550895\tvalid_0's auc: 0.761995\n",
      "[4]\tvalid_0's binary_logloss: 0.524251\tvalid_0's auc: 0.766984\n",
      "[5]\tvalid_0's binary_logloss: 0.504433\tvalid_0's auc: 0.770568\n",
      "[6]\tvalid_0's binary_logloss: 0.489087\tvalid_0's auc: 0.770545\n",
      "[7]\tvalid_0's binary_logloss: 0.47757\tvalid_0's auc: 0.770446\n",
      "[8]\tvalid_0's binary_logloss: 0.468269\tvalid_0's auc: 0.770626\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[10]\tvalid_0's binary_logloss: 0.455769\tvalid_0's auc: 0.769739\n",
      "[11]\tvalid_0's binary_logloss: 0.451533\tvalid_0's auc: 0.769676\n",
      "[12]\tvalid_0's binary_logloss: 0.448643\tvalid_0's auc: 0.769198\n",
      "[13]\tvalid_0's binary_logloss: 0.446163\tvalid_0's auc: 0.77009\n",
      "[14]\tvalid_0's binary_logloss: 0.444737\tvalid_0's auc: 0.769141\n",
      "Early stopping, best iteration is:\n",
      "[9]\tvalid_0's binary_logloss: 0.460986\tvalid_0's auc: 0.770883\n",
      "[1]\tvalid_0's binary_logloss: 0.630622\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585383\tvalid_0's auc: 0.755866\n",
      "[3]\tvalid_0's binary_logloss: 0.550719\tvalid_0's auc: 0.762993\n",
      "[4]\tvalid_0's binary_logloss: 0.524539\tvalid_0's auc: 0.764337\n",
      "[5]\tvalid_0's binary_logloss: 0.504647\tvalid_0's auc: 0.765675\n",
      "[6]\tvalid_0's binary_logloss: 0.489179\tvalid_0's auc: 0.76446\n",
      "[7]\tvalid_0's binary_logloss: 0.47741\tvalid_0's auc: 0.765702\n",
      "[8]\tvalid_0's binary_logloss: 0.468487\tvalid_0's auc: 0.765607\n",
      "[9]\tvalid_0's binary_logloss: 0.461515\tvalid_0's auc: 0.766937\n",
      "[10]\tvalid_0's binary_logloss: 0.455857\tvalid_0's auc: 0.767589\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[12]\tvalid_0's binary_logloss: 0.448367\tvalid_0's auc: 0.768973\n",
      "[13]\tvalid_0's binary_logloss: 0.446381\tvalid_0's auc: 0.768027\n",
      "[14]\tvalid_0's binary_logloss: 0.444324\tvalid_0's auc: 0.768433\n",
      "[15]\tvalid_0's binary_logloss: 0.443134\tvalid_0's auc: 0.768511\n",
      "[16]\tvalid_0's binary_logloss: 0.442375\tvalid_0's auc: 0.767896\n",
      "Early stopping, best iteration is:\n",
      "[11]\tvalid_0's binary_logloss: 0.451023\tvalid_0's auc: 0.770397\n",
      "[1]\tvalid_0's binary_logloss: 0.622178\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.572399\tvalid_0's auc: 0.75316\n",
      "[3]\tvalid_0's binary_logloss: 0.537466\tvalid_0's auc: 0.758069\n",
      "[4]\tvalid_0's binary_logloss: 0.511338\tvalid_0's auc: 0.762643\n",
      "[5]\tvalid_0's binary_logloss: 0.492212\tvalid_0's auc: 0.764072\n",
      "[6]\tvalid_0's binary_logloss: 0.478392\tvalid_0's auc: 0.763836\n",
      "[7]\tvalid_0's binary_logloss: 0.467832\tvalid_0's auc: 0.765215\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[9]\tvalid_0's binary_logloss: 0.45493\tvalid_0's auc: 0.765335\n",
      "[10]\tvalid_0's binary_logloss: 0.450702\tvalid_0's auc: 0.766173\n",
      "[11]\tvalid_0's binary_logloss: 0.44805\tvalid_0's auc: 0.766415\n",
      "[12]\tvalid_0's binary_logloss: 0.445777\tvalid_0's auc: 0.76658\n",
      "[13]\tvalid_0's binary_logloss: 0.444291\tvalid_0's auc: 0.766313\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[1]\tvalid_0's binary_logloss: 0.621248\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.571859\tvalid_0's auc: 0.755741\n",
      "[3]\tvalid_0's binary_logloss: 0.535923\tvalid_0's auc: 0.7616\n",
      "[4]\tvalid_0's binary_logloss: 0.510586\tvalid_0's auc: 0.764651\n",
      "[5]\tvalid_0's binary_logloss: 0.491362\tvalid_0's auc: 0.767661\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[7]\tvalid_0's binary_logloss: 0.467254\tvalid_0's auc: 0.769386\n",
      "[8]\tvalid_0's binary_logloss: 0.459963\tvalid_0's auc: 0.768808\n",
      "[9]\tvalid_0's binary_logloss: 0.454337\tvalid_0's auc: 0.767822\n",
      "[10]\tvalid_0's binary_logloss: 0.450393\tvalid_0's auc: 0.767322\n",
      "[11]\tvalid_0's binary_logloss: 0.447231\tvalid_0's auc: 0.768387\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[1]\tvalid_0's binary_logloss: 0.621209\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.57127\tvalid_0's auc: 0.762518\n",
      "[3]\tvalid_0's binary_logloss: 0.53593\tvalid_0's auc: 0.765924\n",
      "[4]\tvalid_0's binary_logloss: 0.509763\tvalid_0's auc: 0.765695\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[6]\tvalid_0's binary_logloss: 0.476556\tvalid_0's auc: 0.766273\n",
      "[7]\tvalid_0's binary_logloss: 0.466645\tvalid_0's auc: 0.765305\n",
      "[8]\tvalid_0's binary_logloss: 0.45997\tvalid_0's auc: 0.763679\n",
      "[9]\tvalid_0's binary_logloss: 0.454735\tvalid_0's auc: 0.763678\n",
      "[10]\tvalid_0's binary_logloss: 0.450864\tvalid_0's auc: 0.764381\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[1]\tvalid_0's binary_logloss: 0.622178\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.572399\tvalid_0's auc: 0.75316\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\tvalid_0's binary_logloss: 0.537466\tvalid_0's auc: 0.758069\n",
      "[4]\tvalid_0's binary_logloss: 0.511338\tvalid_0's auc: 0.762643\n",
      "[5]\tvalid_0's binary_logloss: 0.492212\tvalid_0's auc: 0.764072\n",
      "[6]\tvalid_0's binary_logloss: 0.478392\tvalid_0's auc: 0.763836\n",
      "[7]\tvalid_0's binary_logloss: 0.467832\tvalid_0's auc: 0.765215\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[9]\tvalid_0's binary_logloss: 0.45493\tvalid_0's auc: 0.765335\n",
      "[10]\tvalid_0's binary_logloss: 0.450702\tvalid_0's auc: 0.766173\n",
      "[11]\tvalid_0's binary_logloss: 0.44805\tvalid_0's auc: 0.766415\n",
      "[12]\tvalid_0's binary_logloss: 0.445777\tvalid_0's auc: 0.76658\n",
      "[13]\tvalid_0's binary_logloss: 0.444291\tvalid_0's auc: 0.766313\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[1]\tvalid_0's binary_logloss: 0.621248\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.571859\tvalid_0's auc: 0.755741\n",
      "[3]\tvalid_0's binary_logloss: 0.535923\tvalid_0's auc: 0.7616\n",
      "[4]\tvalid_0's binary_logloss: 0.510586\tvalid_0's auc: 0.764651\n",
      "[5]\tvalid_0's binary_logloss: 0.491362\tvalid_0's auc: 0.767661\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[7]\tvalid_0's binary_logloss: 0.467254\tvalid_0's auc: 0.769386\n",
      "[8]\tvalid_0's binary_logloss: 0.459963\tvalid_0's auc: 0.768808\n",
      "[9]\tvalid_0's binary_logloss: 0.454337\tvalid_0's auc: 0.767822\n",
      "[10]\tvalid_0's binary_logloss: 0.450393\tvalid_0's auc: 0.767322\n",
      "[11]\tvalid_0's binary_logloss: 0.447231\tvalid_0's auc: 0.768387\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[1]\tvalid_0's binary_logloss: 0.621209\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.57127\tvalid_0's auc: 0.762518\n",
      "[3]\tvalid_0's binary_logloss: 0.53593\tvalid_0's auc: 0.765924\n",
      "[4]\tvalid_0's binary_logloss: 0.509763\tvalid_0's auc: 0.765695\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[6]\tvalid_0's binary_logloss: 0.476556\tvalid_0's auc: 0.766273\n",
      "[7]\tvalid_0's binary_logloss: 0.466645\tvalid_0's auc: 0.765305\n",
      "[8]\tvalid_0's binary_logloss: 0.45997\tvalid_0's auc: 0.763679\n",
      "[9]\tvalid_0's binary_logloss: 0.454735\tvalid_0's auc: 0.763678\n",
      "[10]\tvalid_0's binary_logloss: 0.450864\tvalid_0's auc: 0.764381\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[1]\tvalid_0's binary_logloss: 0.622178\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.572399\tvalid_0's auc: 0.75316\n",
      "[3]\tvalid_0's binary_logloss: 0.537466\tvalid_0's auc: 0.758069\n",
      "[4]\tvalid_0's binary_logloss: 0.511338\tvalid_0's auc: 0.762643\n",
      "[5]\tvalid_0's binary_logloss: 0.492212\tvalid_0's auc: 0.764072\n",
      "[6]\tvalid_0's binary_logloss: 0.478392\tvalid_0's auc: 0.763836\n",
      "[7]\tvalid_0's binary_logloss: 0.467832\tvalid_0's auc: 0.765215\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[9]\tvalid_0's binary_logloss: 0.45493\tvalid_0's auc: 0.765335\n",
      "[10]\tvalid_0's binary_logloss: 0.450702\tvalid_0's auc: 0.766173\n",
      "[11]\tvalid_0's binary_logloss: 0.44805\tvalid_0's auc: 0.766415\n",
      "[12]\tvalid_0's binary_logloss: 0.445777\tvalid_0's auc: 0.76658\n",
      "[13]\tvalid_0's binary_logloss: 0.444291\tvalid_0's auc: 0.766313\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[1]\tvalid_0's binary_logloss: 0.621248\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.571859\tvalid_0's auc: 0.755741\n",
      "[3]\tvalid_0's binary_logloss: 0.535923\tvalid_0's auc: 0.7616\n",
      "[4]\tvalid_0's binary_logloss: 0.510586\tvalid_0's auc: 0.764651\n",
      "[5]\tvalid_0's binary_logloss: 0.491362\tvalid_0's auc: 0.767661\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[7]\tvalid_0's binary_logloss: 0.467254\tvalid_0's auc: 0.769386\n",
      "[8]\tvalid_0's binary_logloss: 0.459963\tvalid_0's auc: 0.768808\n",
      "[9]\tvalid_0's binary_logloss: 0.454337\tvalid_0's auc: 0.767822\n",
      "[10]\tvalid_0's binary_logloss: 0.450393\tvalid_0's auc: 0.767322\n",
      "[11]\tvalid_0's binary_logloss: 0.447231\tvalid_0's auc: 0.768387\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[1]\tvalid_0's binary_logloss: 0.621209\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.57127\tvalid_0's auc: 0.762518\n",
      "[3]\tvalid_0's binary_logloss: 0.53593\tvalid_0's auc: 0.765924\n",
      "[4]\tvalid_0's binary_logloss: 0.509763\tvalid_0's auc: 0.765695\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[6]\tvalid_0's binary_logloss: 0.476556\tvalid_0's auc: 0.766273\n",
      "[7]\tvalid_0's binary_logloss: 0.466645\tvalid_0's auc: 0.765305\n",
      "[8]\tvalid_0's binary_logloss: 0.45997\tvalid_0's auc: 0.763679\n",
      "[9]\tvalid_0's binary_logloss: 0.454735\tvalid_0's auc: 0.763678\n",
      "[10]\tvalid_0's binary_logloss: 0.450864\tvalid_0's auc: 0.764381\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[1]\tvalid_0's binary_logloss: 0.622178\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.572399\tvalid_0's auc: 0.75316\n",
      "[3]\tvalid_0's binary_logloss: 0.537466\tvalid_0's auc: 0.758069\n",
      "[4]\tvalid_0's binary_logloss: 0.511338\tvalid_0's auc: 0.762643\n",
      "[5]\tvalid_0's binary_logloss: 0.492212\tvalid_0's auc: 0.764072\n",
      "[6]\tvalid_0's binary_logloss: 0.478392\tvalid_0's auc: 0.763836\n",
      "[7]\tvalid_0's binary_logloss: 0.467832\tvalid_0's auc: 0.765215\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[9]\tvalid_0's binary_logloss: 0.45493\tvalid_0's auc: 0.765335\n",
      "[10]\tvalid_0's binary_logloss: 0.450702\tvalid_0's auc: 0.766173\n",
      "[11]\tvalid_0's binary_logloss: 0.44805\tvalid_0's auc: 0.766415\n",
      "[12]\tvalid_0's binary_logloss: 0.445777\tvalid_0's auc: 0.76658\n",
      "[13]\tvalid_0's binary_logloss: 0.444291\tvalid_0's auc: 0.766313\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[1]\tvalid_0's binary_logloss: 0.621248\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.571859\tvalid_0's auc: 0.755741\n",
      "[3]\tvalid_0's binary_logloss: 0.535923\tvalid_0's auc: 0.7616\n",
      "[4]\tvalid_0's binary_logloss: 0.510586\tvalid_0's auc: 0.764651\n",
      "[5]\tvalid_0's binary_logloss: 0.491362\tvalid_0's auc: 0.767661\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[7]\tvalid_0's binary_logloss: 0.467254\tvalid_0's auc: 0.769386\n",
      "[8]\tvalid_0's binary_logloss: 0.459963\tvalid_0's auc: 0.768808\n",
      "[9]\tvalid_0's binary_logloss: 0.454337\tvalid_0's auc: 0.767822\n",
      "[10]\tvalid_0's binary_logloss: 0.450393\tvalid_0's auc: 0.767322\n",
      "[11]\tvalid_0's binary_logloss: 0.447231\tvalid_0's auc: 0.768387\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[1]\tvalid_0's binary_logloss: 0.621209\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.57127\tvalid_0's auc: 0.762518\n",
      "[3]\tvalid_0's binary_logloss: 0.53593\tvalid_0's auc: 0.765924\n",
      "[4]\tvalid_0's binary_logloss: 0.509763\tvalid_0's auc: 0.765695\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[6]\tvalid_0's binary_logloss: 0.476556\tvalid_0's auc: 0.766273\n",
      "[7]\tvalid_0's binary_logloss: 0.466645\tvalid_0's auc: 0.765305\n",
      "[8]\tvalid_0's binary_logloss: 0.45997\tvalid_0's auc: 0.763679\n",
      "[9]\tvalid_0's binary_logloss: 0.454735\tvalid_0's auc: 0.763678\n",
      "[10]\tvalid_0's binary_logloss: 0.450864\tvalid_0's auc: 0.764381\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[1]\tvalid_0's binary_logloss: 0.622178\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.572399\tvalid_0's auc: 0.75316\n",
      "[3]\tvalid_0's binary_logloss: 0.537466\tvalid_0's auc: 0.758069\n",
      "[4]\tvalid_0's binary_logloss: 0.511338\tvalid_0's auc: 0.762643\n",
      "[5]\tvalid_0's binary_logloss: 0.492212\tvalid_0's auc: 0.764072\n",
      "[6]\tvalid_0's binary_logloss: 0.478392\tvalid_0's auc: 0.763836\n",
      "[7]\tvalid_0's binary_logloss: 0.467832\tvalid_0's auc: 0.765215\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[9]\tvalid_0's binary_logloss: 0.45493\tvalid_0's auc: 0.765335\n",
      "[10]\tvalid_0's binary_logloss: 0.450702\tvalid_0's auc: 0.766173\n",
      "[11]\tvalid_0's binary_logloss: 0.44805\tvalid_0's auc: 0.766415\n",
      "[12]\tvalid_0's binary_logloss: 0.445777\tvalid_0's auc: 0.76658\n",
      "[13]\tvalid_0's binary_logloss: 0.444291\tvalid_0's auc: 0.766313\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[1]\tvalid_0's binary_logloss: 0.621248\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.571859\tvalid_0's auc: 0.755741\n",
      "[3]\tvalid_0's binary_logloss: 0.535923\tvalid_0's auc: 0.7616\n",
      "[4]\tvalid_0's binary_logloss: 0.510586\tvalid_0's auc: 0.764651\n",
      "[5]\tvalid_0's binary_logloss: 0.491362\tvalid_0's auc: 0.767661\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[7]\tvalid_0's binary_logloss: 0.467254\tvalid_0's auc: 0.769386\n",
      "[8]\tvalid_0's binary_logloss: 0.459963\tvalid_0's auc: 0.768808\n",
      "[9]\tvalid_0's binary_logloss: 0.454337\tvalid_0's auc: 0.767822\n",
      "[10]\tvalid_0's binary_logloss: 0.450393\tvalid_0's auc: 0.767322\n",
      "[11]\tvalid_0's binary_logloss: 0.447231\tvalid_0's auc: 0.768387\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.621209\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.57127\tvalid_0's auc: 0.762518\n",
      "[3]\tvalid_0's binary_logloss: 0.53593\tvalid_0's auc: 0.765924\n",
      "[4]\tvalid_0's binary_logloss: 0.509763\tvalid_0's auc: 0.765695\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[6]\tvalid_0's binary_logloss: 0.476556\tvalid_0's auc: 0.766273\n",
      "[7]\tvalid_0's binary_logloss: 0.466645\tvalid_0's auc: 0.765305\n",
      "[8]\tvalid_0's binary_logloss: 0.45997\tvalid_0's auc: 0.763679\n",
      "[9]\tvalid_0's binary_logloss: 0.454735\tvalid_0's auc: 0.763678\n",
      "[10]\tvalid_0's binary_logloss: 0.450864\tvalid_0's auc: 0.764381\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[1]\tvalid_0's binary_logloss: 0.622178\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.572399\tvalid_0's auc: 0.75316\n",
      "[3]\tvalid_0's binary_logloss: 0.537466\tvalid_0's auc: 0.758069\n",
      "[4]\tvalid_0's binary_logloss: 0.511338\tvalid_0's auc: 0.762643\n",
      "[5]\tvalid_0's binary_logloss: 0.492212\tvalid_0's auc: 0.764072\n",
      "[6]\tvalid_0's binary_logloss: 0.478392\tvalid_0's auc: 0.763836\n",
      "[7]\tvalid_0's binary_logloss: 0.467832\tvalid_0's auc: 0.765215\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[9]\tvalid_0's binary_logloss: 0.45493\tvalid_0's auc: 0.765335\n",
      "[10]\tvalid_0's binary_logloss: 0.450702\tvalid_0's auc: 0.766173\n",
      "[11]\tvalid_0's binary_logloss: 0.44805\tvalid_0's auc: 0.766415\n",
      "[12]\tvalid_0's binary_logloss: 0.445777\tvalid_0's auc: 0.76658\n",
      "[13]\tvalid_0's binary_logloss: 0.444291\tvalid_0's auc: 0.766313\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[1]\tvalid_0's binary_logloss: 0.621248\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.571859\tvalid_0's auc: 0.755741\n",
      "[3]\tvalid_0's binary_logloss: 0.535923\tvalid_0's auc: 0.7616\n",
      "[4]\tvalid_0's binary_logloss: 0.510586\tvalid_0's auc: 0.764651\n",
      "[5]\tvalid_0's binary_logloss: 0.491362\tvalid_0's auc: 0.767661\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[7]\tvalid_0's binary_logloss: 0.467254\tvalid_0's auc: 0.769386\n",
      "[8]\tvalid_0's binary_logloss: 0.459963\tvalid_0's auc: 0.768808\n",
      "[9]\tvalid_0's binary_logloss: 0.454337\tvalid_0's auc: 0.767822\n",
      "[10]\tvalid_0's binary_logloss: 0.450393\tvalid_0's auc: 0.767322\n",
      "[11]\tvalid_0's binary_logloss: 0.447231\tvalid_0's auc: 0.768387\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[1]\tvalid_0's binary_logloss: 0.621209\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.57127\tvalid_0's auc: 0.762518\n",
      "[3]\tvalid_0's binary_logloss: 0.53593\tvalid_0's auc: 0.765924\n",
      "[4]\tvalid_0's binary_logloss: 0.509763\tvalid_0's auc: 0.765695\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[6]\tvalid_0's binary_logloss: 0.476556\tvalid_0's auc: 0.766273\n",
      "[7]\tvalid_0's binary_logloss: 0.466645\tvalid_0's auc: 0.765305\n",
      "[8]\tvalid_0's binary_logloss: 0.45997\tvalid_0's auc: 0.763679\n",
      "[9]\tvalid_0's binary_logloss: 0.454735\tvalid_0's auc: 0.763678\n",
      "[10]\tvalid_0's binary_logloss: 0.450864\tvalid_0's auc: 0.764381\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[1]\tvalid_0's binary_logloss: 0.622178\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.572399\tvalid_0's auc: 0.75316\n",
      "[3]\tvalid_0's binary_logloss: 0.537466\tvalid_0's auc: 0.758069\n",
      "[4]\tvalid_0's binary_logloss: 0.511338\tvalid_0's auc: 0.762643\n",
      "[5]\tvalid_0's binary_logloss: 0.492212\tvalid_0's auc: 0.764072\n",
      "[6]\tvalid_0's binary_logloss: 0.478392\tvalid_0's auc: 0.763836\n",
      "[7]\tvalid_0's binary_logloss: 0.467832\tvalid_0's auc: 0.765215\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[9]\tvalid_0's binary_logloss: 0.45493\tvalid_0's auc: 0.765335\n",
      "[10]\tvalid_0's binary_logloss: 0.450702\tvalid_0's auc: 0.766173\n",
      "[11]\tvalid_0's binary_logloss: 0.44805\tvalid_0's auc: 0.766415\n",
      "[12]\tvalid_0's binary_logloss: 0.445777\tvalid_0's auc: 0.76658\n",
      "[13]\tvalid_0's binary_logloss: 0.444291\tvalid_0's auc: 0.766313\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[1]\tvalid_0's binary_logloss: 0.621248\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.571859\tvalid_0's auc: 0.755741\n",
      "[3]\tvalid_0's binary_logloss: 0.535923\tvalid_0's auc: 0.7616\n",
      "[4]\tvalid_0's binary_logloss: 0.510586\tvalid_0's auc: 0.764651\n",
      "[5]\tvalid_0's binary_logloss: 0.491362\tvalid_0's auc: 0.767661\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[7]\tvalid_0's binary_logloss: 0.467254\tvalid_0's auc: 0.769386\n",
      "[8]\tvalid_0's binary_logloss: 0.459963\tvalid_0's auc: 0.768808\n",
      "[9]\tvalid_0's binary_logloss: 0.454337\tvalid_0's auc: 0.767822\n",
      "[10]\tvalid_0's binary_logloss: 0.450393\tvalid_0's auc: 0.767322\n",
      "[11]\tvalid_0's binary_logloss: 0.447231\tvalid_0's auc: 0.768387\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[1]\tvalid_0's binary_logloss: 0.621209\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.57127\tvalid_0's auc: 0.762518\n",
      "[3]\tvalid_0's binary_logloss: 0.53593\tvalid_0's auc: 0.765924\n",
      "[4]\tvalid_0's binary_logloss: 0.509763\tvalid_0's auc: 0.765695\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[6]\tvalid_0's binary_logloss: 0.476556\tvalid_0's auc: 0.766273\n",
      "[7]\tvalid_0's binary_logloss: 0.466645\tvalid_0's auc: 0.765305\n",
      "[8]\tvalid_0's binary_logloss: 0.45997\tvalid_0's auc: 0.763679\n",
      "[9]\tvalid_0's binary_logloss: 0.454735\tvalid_0's auc: 0.763678\n",
      "[10]\tvalid_0's binary_logloss: 0.450864\tvalid_0's auc: 0.764381\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[1]\tvalid_0's binary_logloss: 0.622178\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.572399\tvalid_0's auc: 0.75316\n",
      "[3]\tvalid_0's binary_logloss: 0.537466\tvalid_0's auc: 0.758069\n",
      "[4]\tvalid_0's binary_logloss: 0.511338\tvalid_0's auc: 0.762643\n",
      "[5]\tvalid_0's binary_logloss: 0.492212\tvalid_0's auc: 0.764072\n",
      "[6]\tvalid_0's binary_logloss: 0.478392\tvalid_0's auc: 0.763836\n",
      "[7]\tvalid_0's binary_logloss: 0.467832\tvalid_0's auc: 0.765215\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[9]\tvalid_0's binary_logloss: 0.45493\tvalid_0's auc: 0.765335\n",
      "[10]\tvalid_0's binary_logloss: 0.450702\tvalid_0's auc: 0.766173\n",
      "[11]\tvalid_0's binary_logloss: 0.44805\tvalid_0's auc: 0.766415\n",
      "[12]\tvalid_0's binary_logloss: 0.445777\tvalid_0's auc: 0.76658\n",
      "[13]\tvalid_0's binary_logloss: 0.444291\tvalid_0's auc: 0.766313\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.459919\tvalid_0's auc: 0.767165\n",
      "[1]\tvalid_0's binary_logloss: 0.621248\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.571859\tvalid_0's auc: 0.755741\n",
      "[3]\tvalid_0's binary_logloss: 0.535923\tvalid_0's auc: 0.7616\n",
      "[4]\tvalid_0's binary_logloss: 0.510586\tvalid_0's auc: 0.764651\n",
      "[5]\tvalid_0's binary_logloss: 0.491362\tvalid_0's auc: 0.767661\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[7]\tvalid_0's binary_logloss: 0.467254\tvalid_0's auc: 0.769386\n",
      "[8]\tvalid_0's binary_logloss: 0.459963\tvalid_0's auc: 0.768808\n",
      "[9]\tvalid_0's binary_logloss: 0.454337\tvalid_0's auc: 0.767822\n",
      "[10]\tvalid_0's binary_logloss: 0.450393\tvalid_0's auc: 0.767322\n",
      "[11]\tvalid_0's binary_logloss: 0.447231\tvalid_0's auc: 0.768387\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.477096\tvalid_0's auc: 0.770393\n",
      "[1]\tvalid_0's binary_logloss: 0.621209\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.57127\tvalid_0's auc: 0.762518\n",
      "[3]\tvalid_0's binary_logloss: 0.53593\tvalid_0's auc: 0.765924\n",
      "[4]\tvalid_0's binary_logloss: 0.509763\tvalid_0's auc: 0.765695\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n",
      "[6]\tvalid_0's binary_logloss: 0.476556\tvalid_0's auc: 0.766273\n",
      "[7]\tvalid_0's binary_logloss: 0.466645\tvalid_0's auc: 0.765305\n",
      "[8]\tvalid_0's binary_logloss: 0.45997\tvalid_0's auc: 0.763679\n",
      "[9]\tvalid_0's binary_logloss: 0.454735\tvalid_0's auc: 0.763678\n",
      "[10]\tvalid_0's binary_logloss: 0.450864\tvalid_0's auc: 0.764381\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.49039\tvalid_0's auc: 0.768033\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.613194\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.560036\tvalid_0's auc: 0.754974\n",
      "[3]\tvalid_0's binary_logloss: 0.524347\tvalid_0's auc: 0.758978\n",
      "[4]\tvalid_0's binary_logloss: 0.499143\tvalid_0's auc: 0.762999\n",
      "[5]\tvalid_0's binary_logloss: 0.481204\tvalid_0's auc: 0.764758\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[7]\tvalid_0's binary_logloss: 0.459945\tvalid_0's auc: 0.76416\n",
      "[8]\tvalid_0's binary_logloss: 0.454027\tvalid_0's auc: 0.764163\n",
      "[9]\tvalid_0's binary_logloss: 0.449603\tvalid_0's auc: 0.765207\n",
      "[10]\tvalid_0's binary_logloss: 0.447169\tvalid_0's auc: 0.764483\n",
      "[11]\tvalid_0's binary_logloss: 0.445318\tvalid_0's auc: 0.764252\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[1]\tvalid_0's binary_logloss: 0.612129\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.559267\tvalid_0's auc: 0.761475\n",
      "[3]\tvalid_0's binary_logloss: 0.523997\tvalid_0's auc: 0.758491\n",
      "[4]\tvalid_0's binary_logloss: 0.497695\tvalid_0's auc: 0.764178\n",
      "[5]\tvalid_0's binary_logloss: 0.480259\tvalid_0's auc: 0.765912\n",
      "[6]\tvalid_0's binary_logloss: 0.467212\tvalid_0's auc: 0.768125\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[8]\tvalid_0's binary_logloss: 0.452239\tvalid_0's auc: 0.768929\n",
      "[9]\tvalid_0's binary_logloss: 0.448669\tvalid_0's auc: 0.768045\n",
      "[10]\tvalid_0's binary_logloss: 0.446528\tvalid_0's auc: 0.766426\n",
      "[11]\tvalid_0's binary_logloss: 0.444016\tvalid_0's auc: 0.76754\n",
      "[12]\tvalid_0's binary_logloss: 0.44255\tvalid_0's auc: 0.767473\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[1]\tvalid_0's binary_logloss: 0.61208\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.558571\tvalid_0's auc: 0.766561\n",
      "[3]\tvalid_0's binary_logloss: 0.522318\tvalid_0's auc: 0.765874\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[5]\tvalid_0's binary_logloss: 0.479211\tvalid_0's auc: 0.76664\n",
      "[6]\tvalid_0's binary_logloss: 0.467124\tvalid_0's auc: 0.766261\n",
      "[7]\tvalid_0's binary_logloss: 0.458312\tvalid_0's auc: 0.76777\n",
      "[8]\tvalid_0's binary_logloss: 0.452781\tvalid_0's auc: 0.765816\n",
      "[9]\tvalid_0's binary_logloss: 0.44809\tvalid_0's auc: 0.766703\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[1]\tvalid_0's binary_logloss: 0.613194\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.560036\tvalid_0's auc: 0.754974\n",
      "[3]\tvalid_0's binary_logloss: 0.524347\tvalid_0's auc: 0.758978\n",
      "[4]\tvalid_0's binary_logloss: 0.499143\tvalid_0's auc: 0.762999\n",
      "[5]\tvalid_0's binary_logloss: 0.481204\tvalid_0's auc: 0.764758\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[7]\tvalid_0's binary_logloss: 0.459945\tvalid_0's auc: 0.76416\n",
      "[8]\tvalid_0's binary_logloss: 0.454027\tvalid_0's auc: 0.764163\n",
      "[9]\tvalid_0's binary_logloss: 0.449603\tvalid_0's auc: 0.765207\n",
      "[10]\tvalid_0's binary_logloss: 0.447169\tvalid_0's auc: 0.764483\n",
      "[11]\tvalid_0's binary_logloss: 0.445318\tvalid_0's auc: 0.764252\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[1]\tvalid_0's binary_logloss: 0.612129\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.559267\tvalid_0's auc: 0.761475\n",
      "[3]\tvalid_0's binary_logloss: 0.523997\tvalid_0's auc: 0.758491\n",
      "[4]\tvalid_0's binary_logloss: 0.497695\tvalid_0's auc: 0.764178\n",
      "[5]\tvalid_0's binary_logloss: 0.480259\tvalid_0's auc: 0.765912\n",
      "[6]\tvalid_0's binary_logloss: 0.467212\tvalid_0's auc: 0.768125\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[8]\tvalid_0's binary_logloss: 0.452239\tvalid_0's auc: 0.768929\n",
      "[9]\tvalid_0's binary_logloss: 0.448669\tvalid_0's auc: 0.768045\n",
      "[10]\tvalid_0's binary_logloss: 0.446528\tvalid_0's auc: 0.766426\n",
      "[11]\tvalid_0's binary_logloss: 0.444016\tvalid_0's auc: 0.76754\n",
      "[12]\tvalid_0's binary_logloss: 0.44255\tvalid_0's auc: 0.767473\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[1]\tvalid_0's binary_logloss: 0.61208\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.558571\tvalid_0's auc: 0.766561\n",
      "[3]\tvalid_0's binary_logloss: 0.522318\tvalid_0's auc: 0.765874\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[5]\tvalid_0's binary_logloss: 0.479211\tvalid_0's auc: 0.76664\n",
      "[6]\tvalid_0's binary_logloss: 0.467124\tvalid_0's auc: 0.766261\n",
      "[7]\tvalid_0's binary_logloss: 0.458312\tvalid_0's auc: 0.76777\n",
      "[8]\tvalid_0's binary_logloss: 0.452781\tvalid_0's auc: 0.765816\n",
      "[9]\tvalid_0's binary_logloss: 0.44809\tvalid_0's auc: 0.766703\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[1]\tvalid_0's binary_logloss: 0.613194\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.560036\tvalid_0's auc: 0.754974\n",
      "[3]\tvalid_0's binary_logloss: 0.524347\tvalid_0's auc: 0.758978\n",
      "[4]\tvalid_0's binary_logloss: 0.499143\tvalid_0's auc: 0.762999\n",
      "[5]\tvalid_0's binary_logloss: 0.481204\tvalid_0's auc: 0.764758\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[7]\tvalid_0's binary_logloss: 0.459945\tvalid_0's auc: 0.76416\n",
      "[8]\tvalid_0's binary_logloss: 0.454027\tvalid_0's auc: 0.764163\n",
      "[9]\tvalid_0's binary_logloss: 0.449603\tvalid_0's auc: 0.765207\n",
      "[10]\tvalid_0's binary_logloss: 0.447169\tvalid_0's auc: 0.764483\n",
      "[11]\tvalid_0's binary_logloss: 0.445318\tvalid_0's auc: 0.764252\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[1]\tvalid_0's binary_logloss: 0.612129\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.559267\tvalid_0's auc: 0.761475\n",
      "[3]\tvalid_0's binary_logloss: 0.523997\tvalid_0's auc: 0.758491\n",
      "[4]\tvalid_0's binary_logloss: 0.497695\tvalid_0's auc: 0.764178\n",
      "[5]\tvalid_0's binary_logloss: 0.480259\tvalid_0's auc: 0.765912\n",
      "[6]\tvalid_0's binary_logloss: 0.467212\tvalid_0's auc: 0.768125\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[8]\tvalid_0's binary_logloss: 0.452239\tvalid_0's auc: 0.768929\n",
      "[9]\tvalid_0's binary_logloss: 0.448669\tvalid_0's auc: 0.768045\n",
      "[10]\tvalid_0's binary_logloss: 0.446528\tvalid_0's auc: 0.766426\n",
      "[11]\tvalid_0's binary_logloss: 0.444016\tvalid_0's auc: 0.76754\n",
      "[12]\tvalid_0's binary_logloss: 0.44255\tvalid_0's auc: 0.767473\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[1]\tvalid_0's binary_logloss: 0.61208\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.558571\tvalid_0's auc: 0.766561\n",
      "[3]\tvalid_0's binary_logloss: 0.522318\tvalid_0's auc: 0.765874\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[5]\tvalid_0's binary_logloss: 0.479211\tvalid_0's auc: 0.76664\n",
      "[6]\tvalid_0's binary_logloss: 0.467124\tvalid_0's auc: 0.766261\n",
      "[7]\tvalid_0's binary_logloss: 0.458312\tvalid_0's auc: 0.76777\n",
      "[8]\tvalid_0's binary_logloss: 0.452781\tvalid_0's auc: 0.765816\n",
      "[9]\tvalid_0's binary_logloss: 0.44809\tvalid_0's auc: 0.766703\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[1]\tvalid_0's binary_logloss: 0.613194\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.560036\tvalid_0's auc: 0.754974\n",
      "[3]\tvalid_0's binary_logloss: 0.524347\tvalid_0's auc: 0.758978\n",
      "[4]\tvalid_0's binary_logloss: 0.499143\tvalid_0's auc: 0.762999\n",
      "[5]\tvalid_0's binary_logloss: 0.481204\tvalid_0's auc: 0.764758\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[7]\tvalid_0's binary_logloss: 0.459945\tvalid_0's auc: 0.76416\n",
      "[8]\tvalid_0's binary_logloss: 0.454027\tvalid_0's auc: 0.764163\n",
      "[9]\tvalid_0's binary_logloss: 0.449603\tvalid_0's auc: 0.765207\n",
      "[10]\tvalid_0's binary_logloss: 0.447169\tvalid_0's auc: 0.764483\n",
      "[11]\tvalid_0's binary_logloss: 0.445318\tvalid_0's auc: 0.764252\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.612129\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.559267\tvalid_0's auc: 0.761475\n",
      "[3]\tvalid_0's binary_logloss: 0.523997\tvalid_0's auc: 0.758491\n",
      "[4]\tvalid_0's binary_logloss: 0.497695\tvalid_0's auc: 0.764178\n",
      "[5]\tvalid_0's binary_logloss: 0.480259\tvalid_0's auc: 0.765912\n",
      "[6]\tvalid_0's binary_logloss: 0.467212\tvalid_0's auc: 0.768125\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[8]\tvalid_0's binary_logloss: 0.452239\tvalid_0's auc: 0.768929\n",
      "[9]\tvalid_0's binary_logloss: 0.448669\tvalid_0's auc: 0.768045\n",
      "[10]\tvalid_0's binary_logloss: 0.446528\tvalid_0's auc: 0.766426\n",
      "[11]\tvalid_0's binary_logloss: 0.444016\tvalid_0's auc: 0.76754\n",
      "[12]\tvalid_0's binary_logloss: 0.44255\tvalid_0's auc: 0.767473\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[1]\tvalid_0's binary_logloss: 0.61208\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.558571\tvalid_0's auc: 0.766561\n",
      "[3]\tvalid_0's binary_logloss: 0.522318\tvalid_0's auc: 0.765874\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[5]\tvalid_0's binary_logloss: 0.479211\tvalid_0's auc: 0.76664\n",
      "[6]\tvalid_0's binary_logloss: 0.467124\tvalid_0's auc: 0.766261\n",
      "[7]\tvalid_0's binary_logloss: 0.458312\tvalid_0's auc: 0.76777\n",
      "[8]\tvalid_0's binary_logloss: 0.452781\tvalid_0's auc: 0.765816\n",
      "[9]\tvalid_0's binary_logloss: 0.44809\tvalid_0's auc: 0.766703\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[1]\tvalid_0's binary_logloss: 0.613194\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.560036\tvalid_0's auc: 0.754974\n",
      "[3]\tvalid_0's binary_logloss: 0.524347\tvalid_0's auc: 0.758978\n",
      "[4]\tvalid_0's binary_logloss: 0.499143\tvalid_0's auc: 0.762999\n",
      "[5]\tvalid_0's binary_logloss: 0.481204\tvalid_0's auc: 0.764758\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[7]\tvalid_0's binary_logloss: 0.459945\tvalid_0's auc: 0.76416\n",
      "[8]\tvalid_0's binary_logloss: 0.454027\tvalid_0's auc: 0.764163\n",
      "[9]\tvalid_0's binary_logloss: 0.449603\tvalid_0's auc: 0.765207\n",
      "[10]\tvalid_0's binary_logloss: 0.447169\tvalid_0's auc: 0.764483\n",
      "[11]\tvalid_0's binary_logloss: 0.445318\tvalid_0's auc: 0.764252\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[1]\tvalid_0's binary_logloss: 0.612129\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.559267\tvalid_0's auc: 0.761475\n",
      "[3]\tvalid_0's binary_logloss: 0.523997\tvalid_0's auc: 0.758491\n",
      "[4]\tvalid_0's binary_logloss: 0.497695\tvalid_0's auc: 0.764178\n",
      "[5]\tvalid_0's binary_logloss: 0.480259\tvalid_0's auc: 0.765912\n",
      "[6]\tvalid_0's binary_logloss: 0.467212\tvalid_0's auc: 0.768125\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[8]\tvalid_0's binary_logloss: 0.452239\tvalid_0's auc: 0.768929\n",
      "[9]\tvalid_0's binary_logloss: 0.448669\tvalid_0's auc: 0.768045\n",
      "[10]\tvalid_0's binary_logloss: 0.446528\tvalid_0's auc: 0.766426\n",
      "[11]\tvalid_0's binary_logloss: 0.444016\tvalid_0's auc: 0.76754\n",
      "[12]\tvalid_0's binary_logloss: 0.44255\tvalid_0's auc: 0.767473\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[1]\tvalid_0's binary_logloss: 0.61208\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.558571\tvalid_0's auc: 0.766561\n",
      "[3]\tvalid_0's binary_logloss: 0.522318\tvalid_0's auc: 0.765874\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[5]\tvalid_0's binary_logloss: 0.479211\tvalid_0's auc: 0.76664\n",
      "[6]\tvalid_0's binary_logloss: 0.467124\tvalid_0's auc: 0.766261\n",
      "[7]\tvalid_0's binary_logloss: 0.458312\tvalid_0's auc: 0.76777\n",
      "[8]\tvalid_0's binary_logloss: 0.452781\tvalid_0's auc: 0.765816\n",
      "[9]\tvalid_0's binary_logloss: 0.44809\tvalid_0's auc: 0.766703\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[1]\tvalid_0's binary_logloss: 0.613194\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.560036\tvalid_0's auc: 0.754974\n",
      "[3]\tvalid_0's binary_logloss: 0.524347\tvalid_0's auc: 0.758978\n",
      "[4]\tvalid_0's binary_logloss: 0.499143\tvalid_0's auc: 0.762999\n",
      "[5]\tvalid_0's binary_logloss: 0.481204\tvalid_0's auc: 0.764758\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[7]\tvalid_0's binary_logloss: 0.459945\tvalid_0's auc: 0.76416\n",
      "[8]\tvalid_0's binary_logloss: 0.454027\tvalid_0's auc: 0.764163\n",
      "[9]\tvalid_0's binary_logloss: 0.449603\tvalid_0's auc: 0.765207\n",
      "[10]\tvalid_0's binary_logloss: 0.447169\tvalid_0's auc: 0.764483\n",
      "[11]\tvalid_0's binary_logloss: 0.445318\tvalid_0's auc: 0.764252\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[1]\tvalid_0's binary_logloss: 0.612129\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.559267\tvalid_0's auc: 0.761475\n",
      "[3]\tvalid_0's binary_logloss: 0.523997\tvalid_0's auc: 0.758491\n",
      "[4]\tvalid_0's binary_logloss: 0.497695\tvalid_0's auc: 0.764178\n",
      "[5]\tvalid_0's binary_logloss: 0.480259\tvalid_0's auc: 0.765912\n",
      "[6]\tvalid_0's binary_logloss: 0.467212\tvalid_0's auc: 0.768125\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[8]\tvalid_0's binary_logloss: 0.452239\tvalid_0's auc: 0.768929\n",
      "[9]\tvalid_0's binary_logloss: 0.448669\tvalid_0's auc: 0.768045\n",
      "[10]\tvalid_0's binary_logloss: 0.446528\tvalid_0's auc: 0.766426\n",
      "[11]\tvalid_0's binary_logloss: 0.444016\tvalid_0's auc: 0.76754\n",
      "[12]\tvalid_0's binary_logloss: 0.44255\tvalid_0's auc: 0.767473\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[1]\tvalid_0's binary_logloss: 0.61208\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.558571\tvalid_0's auc: 0.766561\n",
      "[3]\tvalid_0's binary_logloss: 0.522318\tvalid_0's auc: 0.765874\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[5]\tvalid_0's binary_logloss: 0.479211\tvalid_0's auc: 0.76664\n",
      "[6]\tvalid_0's binary_logloss: 0.467124\tvalid_0's auc: 0.766261\n",
      "[7]\tvalid_0's binary_logloss: 0.458312\tvalid_0's auc: 0.76777\n",
      "[8]\tvalid_0's binary_logloss: 0.452781\tvalid_0's auc: 0.765816\n",
      "[9]\tvalid_0's binary_logloss: 0.44809\tvalid_0's auc: 0.766703\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[1]\tvalid_0's binary_logloss: 0.613194\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.560036\tvalid_0's auc: 0.754974\n",
      "[3]\tvalid_0's binary_logloss: 0.524347\tvalid_0's auc: 0.758978\n",
      "[4]\tvalid_0's binary_logloss: 0.499143\tvalid_0's auc: 0.762999\n",
      "[5]\tvalid_0's binary_logloss: 0.481204\tvalid_0's auc: 0.764758\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[7]\tvalid_0's binary_logloss: 0.459945\tvalid_0's auc: 0.76416\n",
      "[8]\tvalid_0's binary_logloss: 0.454027\tvalid_0's auc: 0.764163\n",
      "[9]\tvalid_0's binary_logloss: 0.449603\tvalid_0's auc: 0.765207\n",
      "[10]\tvalid_0's binary_logloss: 0.447169\tvalid_0's auc: 0.764483\n",
      "[11]\tvalid_0's binary_logloss: 0.445318\tvalid_0's auc: 0.764252\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[1]\tvalid_0's binary_logloss: 0.612129\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.559267\tvalid_0's auc: 0.761475\n",
      "[3]\tvalid_0's binary_logloss: 0.523997\tvalid_0's auc: 0.758491\n",
      "[4]\tvalid_0's binary_logloss: 0.497695\tvalid_0's auc: 0.764178\n",
      "[5]\tvalid_0's binary_logloss: 0.480259\tvalid_0's auc: 0.765912\n",
      "[6]\tvalid_0's binary_logloss: 0.467212\tvalid_0's auc: 0.768125\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[8]\tvalid_0's binary_logloss: 0.452239\tvalid_0's auc: 0.768929\n",
      "[9]\tvalid_0's binary_logloss: 0.448669\tvalid_0's auc: 0.768045\n",
      "[10]\tvalid_0's binary_logloss: 0.446528\tvalid_0's auc: 0.766426\n",
      "[11]\tvalid_0's binary_logloss: 0.444016\tvalid_0's auc: 0.76754\n",
      "[12]\tvalid_0's binary_logloss: 0.44255\tvalid_0's auc: 0.767473\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[1]\tvalid_0's binary_logloss: 0.61208\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.558571\tvalid_0's auc: 0.766561\n",
      "[3]\tvalid_0's binary_logloss: 0.522318\tvalid_0's auc: 0.765874\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[5]\tvalid_0's binary_logloss: 0.479211\tvalid_0's auc: 0.76664\n",
      "[6]\tvalid_0's binary_logloss: 0.467124\tvalid_0's auc: 0.766261\n",
      "[7]\tvalid_0's binary_logloss: 0.458312\tvalid_0's auc: 0.76777\n",
      "[8]\tvalid_0's binary_logloss: 0.452781\tvalid_0's auc: 0.765816\n",
      "[9]\tvalid_0's binary_logloss: 0.44809\tvalid_0's auc: 0.766703\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.613194\tvalid_0's auc: 0.749019\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.560036\tvalid_0's auc: 0.754974\n",
      "[3]\tvalid_0's binary_logloss: 0.524347\tvalid_0's auc: 0.758978\n",
      "[4]\tvalid_0's binary_logloss: 0.499143\tvalid_0's auc: 0.762999\n",
      "[5]\tvalid_0's binary_logloss: 0.481204\tvalid_0's auc: 0.764758\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[7]\tvalid_0's binary_logloss: 0.459945\tvalid_0's auc: 0.76416\n",
      "[8]\tvalid_0's binary_logloss: 0.454027\tvalid_0's auc: 0.764163\n",
      "[9]\tvalid_0's binary_logloss: 0.449603\tvalid_0's auc: 0.765207\n",
      "[10]\tvalid_0's binary_logloss: 0.447169\tvalid_0's auc: 0.764483\n",
      "[11]\tvalid_0's binary_logloss: 0.445318\tvalid_0's auc: 0.764252\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.468253\tvalid_0's auc: 0.765914\n",
      "[1]\tvalid_0's binary_logloss: 0.612129\tvalid_0's auc: 0.754322\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.559267\tvalid_0's auc: 0.761475\n",
      "[3]\tvalid_0's binary_logloss: 0.523997\tvalid_0's auc: 0.758491\n",
      "[4]\tvalid_0's binary_logloss: 0.497695\tvalid_0's auc: 0.764178\n",
      "[5]\tvalid_0's binary_logloss: 0.480259\tvalid_0's auc: 0.765912\n",
      "[6]\tvalid_0's binary_logloss: 0.467212\tvalid_0's auc: 0.768125\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[8]\tvalid_0's binary_logloss: 0.452239\tvalid_0's auc: 0.768929\n",
      "[9]\tvalid_0's binary_logloss: 0.448669\tvalid_0's auc: 0.768045\n",
      "[10]\tvalid_0's binary_logloss: 0.446528\tvalid_0's auc: 0.766426\n",
      "[11]\tvalid_0's binary_logloss: 0.444016\tvalid_0's auc: 0.76754\n",
      "[12]\tvalid_0's binary_logloss: 0.44255\tvalid_0's auc: 0.767473\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.458183\tvalid_0's auc: 0.769644\n",
      "[1]\tvalid_0's binary_logloss: 0.61208\tvalid_0's auc: 0.754597\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.558571\tvalid_0's auc: 0.766561\n",
      "[3]\tvalid_0's binary_logloss: 0.522318\tvalid_0's auc: 0.765874\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[5]\tvalid_0's binary_logloss: 0.479211\tvalid_0's auc: 0.76664\n",
      "[6]\tvalid_0's binary_logloss: 0.467124\tvalid_0's auc: 0.766261\n",
      "[7]\tvalid_0's binary_logloss: 0.458312\tvalid_0's auc: 0.76777\n",
      "[8]\tvalid_0's binary_logloss: 0.452781\tvalid_0's auc: 0.765816\n",
      "[9]\tvalid_0's binary_logloss: 0.44809\tvalid_0's auc: 0.766703\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.496406\tvalid_0's auc: 0.768393\n",
      "[1]\tvalid_0's binary_logloss: 0.631025\tvalid_0's auc: 0.757075\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585192\tvalid_0's auc: 0.768325\n",
      "[3]\tvalid_0's binary_logloss: 0.550885\tvalid_0's auc: 0.770201\n",
      "[4]\tvalid_0's binary_logloss: 0.524532\tvalid_0's auc: 0.772278\n",
      "[5]\tvalid_0's binary_logloss: 0.504805\tvalid_0's auc: 0.771984\n",
      "[6]\tvalid_0's binary_logloss: 0.489337\tvalid_0's auc: 0.771526\n",
      "[7]\tvalid_0's binary_logloss: 0.477446\tvalid_0's auc: 0.771177\n",
      "[8]\tvalid_0's binary_logloss: 0.468529\tvalid_0's auc: 0.770399\n",
      "[9]\tvalid_0's binary_logloss: 0.46168\tvalid_0's auc: 0.769633\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.524532\tvalid_0's auc: 0.772278\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv='warn', error_score='raise-deprecating',\n",
       "       estimator=LGBMClassifier(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,\n",
       "        learning_rate=0.125, max_depth=-1, metric='l1',\n",
       "        min_child_samples=20, min_child_weight=0.001, min_split_gain=0.0,\n",
       "        n_estimators=20, n_jobs=-1, num_leaves=38, objective=None,\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=True,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=1),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'n_estimators': [24, 26, 28, 30, 32, 34, 36, 38], 'learning_rate': [0.1, 0.125, 0.15, 0.175, 0.2]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimator = lgb.LGBMClassifier(learning_rate = 0.125, metric = 'l1', \n",
    "                        n_estimators = 20, num_leaves = 38)\n",
    "\n",
    "\n",
    "param_grid = {\n",
    "    'n_estimators': [x for x in range(24,40,2)],\n",
    "    'learning_rate': [0.10, 0.125, 0.15, 0.175, 0.2]}\n",
    "gridsearch = GridSearchCV(estimator, param_grid)\n",
    "\n",
    "gridsearch.fit(X_train, y_train,\n",
    "        eval_set = [(X_test, y_test)],\n",
    "        eval_metric = ['auc', 'binary_logloss'],\n",
    "        early_stopping_rounds = 5)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters found by grid search are: {'learning_rate': 0.15, 'n_estimators': 24}\n"
     ]
    }
   ],
   "source": [
    "print('Best parameters found by grid search are:', gridsearch.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>LightGBM Hyperparameters + early stopping</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.631082\tvalid_0's auc: 0.756716\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.585608\tvalid_0's auc: 0.763123\n",
      "[3]\tvalid_0's binary_logloss: 0.551121\tvalid_0's auc: 0.768216\n",
      "[4]\tvalid_0's binary_logloss: 0.525003\tvalid_0's auc: 0.768646\n",
      "[5]\tvalid_0's binary_logloss: 0.505251\tvalid_0's auc: 0.768391\n",
      "[6]\tvalid_0's binary_logloss: 0.49003\tvalid_0's auc: 0.768008\n",
      "[7]\tvalid_0's binary_logloss: 0.4781\tvalid_0's auc: 0.768002\n",
      "[8]\tvalid_0's binary_logloss: 0.469401\tvalid_0's auc: 0.767116\n",
      "[9]\tvalid_0's binary_logloss: 0.461946\tvalid_0's auc: 0.768109\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.525003\tvalid_0's auc: 0.768646\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,\n",
       "        learning_rate=0.15, max_depth=-1, metric='l1',\n",
       "        min_child_samples=20, min_child_weight=0.001, min_split_gain=0.0,\n",
       "        n_estimators=24, n_jobs=-1, num_leaves=31, objective=None,\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=True,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=1)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "gbm = lgb.LGBMClassifier(learning_rate = 0.15, metric = 'l1', \n",
    "                        n_estimators = 24)\n",
    "\n",
    "\n",
    "gbm.fit(X_train, y_train,\n",
    "        eval_set=[(X_test, y_test)],\n",
    "        eval_metric=['auc', 'binary_logloss'],\n",
    "early_stopping_rounds=5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Feature Importances Graph </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApYAAAGDCAYAAACRAVC/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3X+cVVW9//HX2wESGX+kA4Yo4ogJOuCg3bRS7yhfDH/UlRsmI11F9KLZD6mvIWWX8N660g9/p90w0syblKhl4iW96lFMyxAhLEW+6qTmb8F0YBAYPt8/9h46HOfHGTgzZ86Z9/PxmAdnr73O2p991j8f1tp7LUUEZmZmZmbba4diB2BmZmZm5cGJpZmZmZkVhBNLMzMzMysIJ5ZmZmZmVhBOLM3MzMysIJxYmpmZmVlBOLE0M+thJP2XpH8rdhxmZp0lr2NpZuVCUgOwJ9CcVfzBiHhpO9qsA26KiL23L7rSJOkG4MWI+HqxYzGzns8jlmZWbj4REZVZf9ucVBaCpD7FvP72kFRR7BjMrLQ4sTSzXkHSEZIelvSWpOXpSGTLuTMlPSnpHUnPSjonLR8A/A+wl6TG9G8vSTdI+mbW9+skvZh13CDpQkl/BNZK6pN+71ZJr0t6TtIX24l1S/stbUuaIek1SS9LOlnSCZKelrRa0teyvjtb0gJJP0/vZ6mkQ7LOj5SUSX+HP0n6ZM51fyDpLklrgbOAycCM9N5/ndabKemZtP0/S5qQ1cYUSQ9J+p6kNem9Hp91fndJ10t6KT3/y6xzJ0lalsb2sKTReXewmfUITizNrOxJGgIsBL4J7A5cANwqaWBa5TXgJGAX4EzgckmHRsRa4HjgpW0YAa0HTgR2AzYDvwaWA0OAscB0SR/Ps60PADum350FXAd8BjgMOAqYJak6q/4/Abek9/oz4JeS+krqm8ZxNzAI+ALw35IOzPruacC3gJ2BG4H/Br6T3vsn0jrPpNfdFbgYuEnS4Kw2DgdWAlXAd4B5kpSe+ymwE3BwGsPlAJIOBX4MnAPsAfwQuEPS+/L8jcysB3BiaWbl5pfpiNdbWaNhnwHuioi7ImJzRNwDLAFOAIiIhRHxTCQeIEm8jtrOOK6KiBciogn4B2BgRPx7RGyIiGdJksNJeba1EfhWRGwE5pMkbFdGxDsR8SfgT0D26N5jEbEgrX8ZSVJ6RPpXCcxJ47gPuJMkCW7xq4j4bfo7rW8tmIi4JSJeSuv8HFgFfDiryl8i4rqIaAZ+AgwG9kyTz+OBcyNiTURsTH9vgH8FfhgRv4+I5oj4CfBuGrOZlYiSffbHzKwNJ0fE/+aU7QucIukTWWV9gfsB0qnabwAfJPkP907Aiu2M44Wc6+8l6a2ssgpgcZ5tvZkmaQBN6b+vZp1vIkkY33PtiNicTtPv1XIuIjZn1f0LyUhoa3G3StLpwJeBYWlRJUmy2+KVrOuvSwcrK0lGUFdHxJpWmt0XOEPSF7LK+mXFbWYlwImlmfUGLwA/jYh/zT2RTrXeCpxOMlq3MR3pbJm6bW3pjLUkyWeLD7RSJ/t7LwDPRcQB2xL8Ntin5YOkHYC9gZYp/H0k7ZCVXA4Fns76bu79bnUsaV+S0daxwCMR0SxpGX//vdrzArC7pN0i4q1Wzn0rIr6VRztm1kN5KtzMeoObgE9I+rikCkk7pi/F7E0yKvY+4HVgUzp6eVzWd18F9pC0a1bZMuCE9EWUDwDTO7j+o8Db6Qs9/dMYaiT9Q8HucGuHSfrn9I306SRTyr8Dfk+SFM9In7msAz5BMr3elleB7Oc3B5Akm69D8uITUJNPUBHxMsnLUNdKen8aw9Hp6euAcyUdrsQASSdK2jnPezazHsCJpZmVvYh4geSFlq+RJEQvAF8BdoiId4AvAr8A1pC8vHJH1nefAm4Gnk2f29yL5AWU5UADyfOYP+/g+s0kCVwt8BzwBvAjkpdfusKvgFNJ7udfgH9On2fcAHyS5DnHN4BrgdPTe2zLPOCglmdWI+LPwKXAIyRJ5yjgt52I7V9Inhl9iuSlqekAEbGE5DnL76dx/z9gSifaNbMewAukm5mVEUmzgeER8Zlix2JmvY9HLM3MzMysIJxYmpmZmVlBeCrczMzMzArCI5ZmZmZmVhBOLM3MzMysILxAehfabbfdYvjw4cUOw7bR2rVrGTBgQLHDsG3k/itd7rvS5v4rXY899tgbETFwe9pwYtmF9txzT5YsWVLsMGwbZTIZ6urqih2GbSP3X+ly35U291/pkvSX7W3DU+FmZmZmVhBOLM3MzMysIJxYmpmZmVlBOLE0MzMzs4JwYmlmZmZmBeHE0szMzMwKwomlmZmZmRWEE0szMzMzKwgnlmZmZmZWEE4szczMzKwgnFiamZmZWUE4sTQzMzOzgnBiaWZmZmYF4cTSzMzMzArCiaWZmZmZFYQTSzMzMzMrCCeWZmZmZlYQTizNzMzMrCCcWJqZmZlZQTixNDMzM7OCcGJpZmZmZgXhxNLMzMzMCsKJpZmZmZkVhBNLMzMzMysIJ5ZmZmZmVhBOLM3MzMysIJxYmpmZmVlBOLE0MzMzs4JwYmlmZmZmBeHE0szMzMwKwomlmZmZmRWEE0szMzMzKwgnlmZmZmZWEE4szczMzKwgnFiamZmZWUE4sTQzMzOzgnBiaWVh6tSpDBo0iJqami1ls2fPZsiQIdTW1lJbW8tdd91VxAjNzMzKX59iB1DOmjY2M2zmwmKHUTYa5pzY5rkpU6bw+c9/ntNPP32r8i996UtccMEFXR2amZmZUeQRS0mNrZTNlnRB+vkGSesk7Zx1/kpJIamqpQ1JoyQtS/9WS3ou/fy/bVx3mKSmtM5ySQ9LOjCnzpWS/ipph6yyKZK+X6j7t8I5+uij2X333YsdhpmZWa9WClPh/w/4J4A0yTsG+Gt2hYhYERG1EVEL3AF8JT3+P+20+0xa5xDgJ8DXWk6k15kAvAAcXdC7sW71/e9/n9GjRzN16lTWrFlT7HDMzMzKWilMhd8MnArcBNQBvwWOL/A1dgGys45jgCeAnwP1QCbfhiRNA6YBVFUNZNaoTYWLspfLZDLtnn/llVdYu3btlnqjR49m3rx5SOLHP/4xp512GhdeeGHe12tsbOzwmtZzuf9Kl/uutLn/erdSSCxXAf8k6f0kSd5NFCax3F/SMmBnYCfg8Kxz9SQJ7a+A/5TUNyI25tNoRMwF5gIMrR4el64ohZ+4NDRMrmv/fEMDAwYMoK7uvfWqq6s56aSTWj3Xlkwm06n61rO4/0qX+660uf96t1KYCge4DZhEkvwtLlCbLVPh+wPTSZNBSf2AE4BfRsTbwO+B4wp0TetGL7/88pbPt99++1ZvjJuZmVnhlcpw2nxgKfCTiNgsqdDt3wFcn34eD+wKrEivsxOwDuj06939+1awsp03ma1w6uvryWQyvPHGG+y9995cfPHFZDIZli1bhiSGDRvGD3/4w2KHaWZmVtZKIrGMiOclXQS0+pZ3ARwJPJN+rgfOjoibASQNAJ6TtFMXXdsK4Oabb35P2VlnnVWESMzMzHqvYieWO0l6Mev4srYqRkShh5tanrEUsAE4O00ePw6ck3XdtZIeAj6RFk2RdHJWO0dERPY9mJmZmfVKRU0sI6LdZzwjYkob5cOyPlfm852cOg1A/zZOv2cxxIj456zDGzpq38zMzKw3KpWXd8zMzMyshyv2VHiXkjQK+GlO8bsRcXhr9c3MzMxs25V1YhkRK4DaYsdhZmZm1ht4Kty61dSpUxk0aNBWa0recsstHHzwweywww4sWbKkiNGZmZnZ9nBiad1qypQpLFq0aKuympoabrvtNo4+2tuym5mZlbKymgqX1AysILmvJ4EzImJdem4CyQ4+IyPiKUk7AsuAU9IpcyTNAKoj4tw22h8K/AjYBwjghPQN81Y1bWxm2MxOr6te8hraWRT+6KOPpqGhYauykSNHdnFEZmZm1h3KbcSyKd2msYZkbcrsBLEeeIhka0giYj3JVo7XKjGEZP3Kr7bT/o3AdyNiJPBh4LUuuAczMzOzklRWI5Y5FgOjASRVAh8DjiHZvnE2QEQskjQVOB04EZgdEWtaa0zSQUCfiLgn/W5jG/WmAdMAqqoGMmvUpgLeUmnIZDLtnn/llVdYu3bte+q99dZbPPbYYzQ2tvrTdrvGxsYO78V6Lvdf6XLflTb3X+9WlomlpD7A8UDLw3wnA4si4mlJqyUdGhFL03PTgUeBVRGRuzRRtg8Cb0m6DdiPZHvJmRHRnF0pIuYCcwGGVg+PS1eU5U/crobJde2fb2hgwIAB1NVtXW+33XbjsMMO40Mf+lDXBdcJmUzmPTFa6XD/lS73XWlz//Vu5TYV3j/dpnEJ8DwwLy2vB+ann+enxwBExEvAfcAPOmi7D3AUcAHwD0A1MKVQgZuZmZmVunIbTmuKiK3WrZS0B3AsUCMpgAogJM2IiEirbU7/2vMi8HhEPJu2+0vgCP6evL5H/74VrGznRZbeqL6+nkwmwxtvvMHee+/NxRdfzO67784XvvAFXn/9dU488URqa2v5zW9+U+xQzczMrJPKLbFszUTgxog4p6VA0gPAkSTPYebrD8D7JQ2MiNdJklUvuthJN998c6vlEyZM6OZIzMzMrNDKbSq8NfXA7TlltwKndaaR9FnKC4B7Ja0ABFxXkAjNzMzMykBZjVhGRGUrZXWtlF2Vczwlz/bvIX3T3MzMzMy21htGLM3MzMysG5TViGUhSBoF5C479G5EHF6MeMzMzMxKhRPLHOn2jrUdVjQzMzOzrXgq3LrE1KlTGTRoEDU1NVvKVq9ezbhx4zjggAMYN24ca9a0usmRmZmZlSgnltYlpkyZwqJFi7YqmzNnDmPHjmXVqlWMHTuWOXPmFCk6MzMz6wrdPhUuqRloWa6nGfh8RDwsaRhwZ0TUSKoDLoiIk3K+m0nL81o/UtKVJOtY7hMRm9OyKcD1wP+JiHvTsgnAbcApwGSSLRsrgYHAc2lz5wGHkmwBuT8wMCLeaO/6TRubGTZzYT6hlqSGdhZ/P/roo2loaNiq7Fe/+tWW/WPPOOMM6urq+Pa3v92FEZqZmVl3KsYzllt2x5H0ceAS4B8LfRFJOwATgBeAo4FM1ukVJOtb3pseTwKWA0TEhPT7deQkt5KagDtz2rI8vfrqqwwePBiAwYMH89prrxU5IjMzMyukYk+F7wJ01YN2xwBPkOwBXp9zbjHwYUl9JVUCw4FlHTUYEY9HREOhAzUzMzMrB8UYsewvaRmwIzCYZGvErlAP3Az8CvhPSX0jYmN6LoD/BT4O7ArcQTL9vd0kTQOmAVRVDWTWqE2FaLZHapnWbssrr7zC2rVrt9TbZZdduPXWW9ljjz1488032XnnnTtso5gaGxt7dHzWPvdf6XLflTb3X+9W7KnwjwA3Sqrp4DudIqkfcALwpYh4R9LvgeOA7Ace5wNfJEks/y/wtUJcOyLmAnMBhlYPj0tXlO+KTg2T69o/39DAgAEDqKtL6p166qmsWrWKT33qU8yZM4dJkyZtOdcTZTKZHh2ftc/9V7rcd6XN/de7FTXriYhHJFWRvCRTSONJEsYVkgB2AtaRlVhGxKNpQtsUEU+n9Qqqf98KVrbzgks5q6+vJ5PJ8MYbb7D33ntz8cUXM3PmTD796U8zb948hg4dyi233FLsMM3MzKyAippYShoBVABvkiR/hVIPnB0RN6fXGQA8Jyn3Gl8F1hfwupa6+eabWy2/9957Wy03MzOz0lfMZywhWXLojIhobmXEcKykF7OOT0n/XSip5VnJRyLilOwvpcnjx4FzWsoiYq2kh4BPZNeNiP/pTOCSvgjMAD4A/FHSXRFxdmfaMDMzMytX3Z5YRkRFG+UNQE36OQP0b6VaXR7trwN2b6X8n7MOb2jl/JSc4ww5ywpFxFXAVR3FYGZmZtYbFXu5ITMzMzMrEyX9ynK6wHru1i3PtSxybmZmZmbdp6QTy4j4DfCbYsdhZmZmZp4Kty4ydepUBg0aRE3N35coXb16NePGjeOAAw5g3LhxrFnTVZsumZmZWTH02sRS0gRJkS551FJ2gKQ7JT0j6TFJ90s6Oj03RdLrkpZl/R1UvDvo2aZMmcKiRYu2KpszZw5jx45l1apVjB07ljlz5hQpOjMzM+sKJT0Vvp3qgYeAScBsSTuSLKB+QUTcAZAuoP4h4MH0Oz+PiM/ne4Gmjc0Mm7mw44olqqGdxd+PPvpoGhoatir71a9+tWWbrzPOOIO6ujq+/e3cR2TNzMysVPXKEUtJlcDHgLNIEkuAySTrYt7RUi8inoiIG7o/wvL06quvMnjwYAAGDx7Ma6+9VuSIzMzMrJB664jlycCidCvH1ZIOBQ4GlnbwvVMlHZl1/JGIaOqyKM3MzMxKSG9NLOuBK9LP89PjrUi6HTgAeDprcfUOp8IlTQOmAVRVDWTWqE0FC7qnaZnWbssrr7zC2rVrt9TbZZdduPXWW9ljjz1488032XnnnTtso5gaGxt7dHzWPvdf6XLflTb3X+/W6xJLSXsAxwI1koJkr/IALgaObqkXERMkfQj4Xmfaj4i5wFyAodXD49IV5fsTN0yua/98QwMDBgygri6pd+qpp7Jq1So+9alPMWfOHCZNmrTlXE+UyWR6dHzWPvdf6XLflTb3X+/WG5+xnAjcGBH7RsSwiNgHeA54GviYpE9m1d2pKBGWgfr6ej7ykY+wcuVK9t57b+bNm8fMmTO55557OOCAA7jnnnuYOXNmscM0MzOzAirf4bS21QO569zcCpwGnARcJukK4FXgHeCbWfVyn7E8LyIebutC/ftWsLKdN6fL2c0339xq+b333tvNkZiZmVl36XWJZUTUtVJ2VdbhCW187wbghi4JyszMzKwM9MapcDMzMzPrAk4szczMzKwgnFiamZmZWUE4sTQzMzOzgnBiadvtyiuvpKamhoMPPpgrrrii4y+YmZlZWXJiadvliSee4LrrruPRRx9l+fLl3HnnnaxatarYYZmZmVkRFG25IUnNwIo0hieBMyJiXXpuAnAbMDIinpK0I7AMOCUiVqR1ZgDVEXFuO9f4EnAJsGdE/C0tqwPuB86OiHlp2RiSfcK/AuwHfAzol35emTb3TeBw4BPABuAZ4MyIeKut6zdtbGbYzIWd/GV6poY21uN88sknOeKII9hpp2Qt+X/8x3/k9ttvZ8aMGd0ZnpmZmfUAxRyxbIqI2oioIUnUshPEeuAhYBJARKwHpgPXKjEEOAf4agfXqAf+AEzIKV8BnJp1PAlYnl7rcxFRS7Ke5TNpjLURsQC4B6iJiNEkO/V0dP2yV1NTw4MPPsibb77JunXruOuuu3jhhReKHZaZmZkVQU+ZCl8MDAeQVEkyYngWaWIJEBGLgJeB04HLgdkRsaatBiXtD1QCXydJMLM9D+woaU9JAsYD/9NRkBFxd0RsSg9/B+yd192VsZEjR3LhhRcybtw4xo8fzyGHHEKfPr1u3X0zMzOjB+y8I6kPcDywKC06GVgUEU9LWi3p0IhYmp6bDjwKrIqIn3bQdD1wM0nSeqCkQRHxWtb5BcApwOMk0+DvdjL0qcDPW7mfacA0gKqqgcwatSm3SknKZDJtntt///257LLLALjuuuvYcccd261fKhobG8viPnor91/pct+VNvdf71bMxLK/pGXp58XAvPRzPdDyavH89HgpQES8JOk+4M482p8ETIiIzZJuI0kir8k6/wuSxHAESQL60XwDl3QRsAn479xzETEXmAswtHp4XLqi6Ll7QTRMrmvz3GuvvcagQYN4/vnneeyxx3jkkUd4//vf333BdZFMJkNdXV2xw7Bt5P4rXe670ub+692KmfU0pc8ybiFpD+BYoEZSABVASJoREZFW25z+tUnSaOAA4J5kppt+wLNkJZYR8YqkjcA44HzyTCwlnQGcBIzNiqlX+9SnPsWbb75J3759ueaaa8oiqTQzM7PO62nDaROBGyPinJYCSQ8AR5KMauarnuQZzEuy2nlO0r459WYBgyKiOU1A2yVpPHAh8I8tb7C3p3/fCla28TZ1OVm8uDNdY2ZmZuWqp7y806IeuD2n7FbgtE62M6mVdm4n62UggIh4OCJ+2Yl2vw/sTDISukzSf3UyLjMzM7OyVbQRy4iobKWsrpWyq3KOp+TR9n6tlH056zDTyvnZOccNQE1O2fCOrm1mZmbWW/W0EUszMzMzK1E97RnLTpE0CshddujdiDi8GPGYmZmZ9WYlnVim2zvWdljRzMzMzLqcp8LNzMzMrCCcWFreLr/8cg4++GBqamqor69n/fr1xQ7JzMzMepCSngrv6Zo2NjNs5sJih9EpDW2su/nXv/6Vq666ij//+c/079+fT3/608yfP58pU6Z0b4BmZmbWYxVtxFJSc7oW5BOSbpG0U9a5CZJC0oj0eEdJT6Uv67TUmdHROpKSviRpvaRds8rq0rbPyiobk5ZdIOmaNK4/S2pKPy+TNFHSf0j6Y3p8t6S9Cvur9GybNm2iqamJTZs2sW7dOvbaq1fdvpmZmXWgmFPhTRFRGxE1wAbg3Kxz9cBDpAuaR8R6YDpwrRJDgHOAr3ZwjXrgD8CEnPIVwKlZx5OA5em1PpduNXkC8EwaY21ELAC+GxGj0/N3kuzc0ysMGTKECy64gKFDhzJ48GB23XVXjjvuuGKHZWZmZj1IT5kKXwyMBpBUCXwMOAa4A5gNEBGLJE0FTgdOJNmycU1bDUraH6gEvgJ8Dbgh6/TzwC6S9gReA8YDd3UUZES8nXU4AHjPXuGSpgHTAKqqBjJr1KaOmu1RMplMq+XvvPMOP/nJT7jpppuorKxk9uzZXHTRRYwbN657A+xGjY2Nbf4e1vO5/0qX+660uf96t6InlpL6AMcDi9Kik4FFEfG0pNWSDo2Ipem56cCjwKqIyF2/Mlc9cDNJ0nqgpEER8VrW+QXAKcDjwFLg3Tzj/RZJcvs3kuR3KxExF5gLMLR6eFy6oug/cac0TK5rtfyWW25hzJgxnHzyyQC89NJL/O53v6OurvX65SCTyZT1/ZU791/pct+VNvdf71bMqfD+kpYBS0hGEOel5fXA/PTz/PQYgIh4CbgP+EEe7U8C5kfEZuA2kiQy2y/SspYENC8RcVFE7AP8N/D5fL9X6oYOHcrvfvc71q1bR0Rw7733MnLkyGKHZWZmZj1IMYfTmtJnFbeQtAdwLFAjKYAKICTNiIiWaefN6V+bJI0GDgDukQTQD3gWuKalTkS8ImkjMA44H/hoJ+P/GbAQ+EZbFfr3rWBlG29Zl5rDDz+ciRMncuihh9KnTx/GjBnDtGnTih2WmZmZ9SA9bZ52InBjRJzTUiDpAeBIkintfNWTPIN5SVY7z0naN6feLGBQRDSnCWi7JB0QEavSw08CT3UippJ38cUXc/HFFxc7DDMzM+uhelpiWQ/MySm7FTiNziWWk0ie28x2e1r++5aCiHi4k/HNkXQgyYjpX9j6TXYzMzOzXq1oiWVEVLZSVtdK2VU5x1PyaHu/Vsq+nHWYaeX87JzjBqAmp+xTHV3bzMzMrLfylo5mZmZmVhA9bSq8U9KdeHKXHXo3Ig4vRjxmZmZmvVlJJ5YRsQKo7bCimZmZmXU5T4X3cm+99RYTJ05kxIgRjBw5kkceeaTYIZmZmVmJKukRy56uaWMzw2YuLHYYNLSzlub555/P+PHjWbBgARs2bGDdunXdGJmZmZmVk6KNWEpqlrRM0hOSbpG0U9a5CZJC0oj0eEdJT6XPVLbUmSHpvzq4xpckrZe0a1ZZXdr2WVllY9KyCyRdk8b1Z0lN6edlkiamdb8gaaWkP0n6TiF/k+729ttv8+CDD3LWWclP0a9fP3bbbbciR2VmZmalqphT4U0RURsRNcAGtl4Tsh54iGTdSSJiPck+4dcqMQQ4B/hqB9eoB/4ATMgpXwGcmnU8CVieXutz6Y5AJwDPpDHWRsQCSccA/wSMjoiDge91+q57kGeffZaBAwdy5plnMmbMGM4++2zWrl1b7LDMzMysRPWUqfDFwGgASZXAx4BjgDuA2QARsUjSVOB04ESSnXXWtNWgpP2BSuArwNeAG7JOPw/sImlP4DVgPHBXHnF+FpgTEe+mMb3WynWnAdMAqqoGMmvUpjya7VqZTKbV8pUrV/LYY48xZcoUpkyZwtVXX81nP/tZpk6d2r0B9lCNjY1t/nbW87n/Spf7rrS5/3q3oieWkvqQ7JKzKC06GVgUEU9LWi3p0IhYmp6bDjwKrIqI3GWGctUDN5MkrQdKGpSTCC4ATgEeB5YC7+YR7geBoyR9C1gPXBARf8iuEBFzgbkAQ6uHx6Uriv4T0zC5rtXyESNGcMkll3DeeecBUFFRwZw5c6ira71+b5PJZPxblDD3X+ly35U291/vVsyp8P6SlgFLSEYQ56Xl9cD89PP89BiAiHgJuA/4QR7tTwLmR8Rm4DaSJDLbL9KylgQ0H32A9wNHkIyE/kL5bDLeQ33gAx9gn332YeXKlQDce++9HHTQQUWOyszMzEpVMYfTmtJnGbeQtAdwLFAjKYAKICTNiIhIq21O/9okaTRwAHBPmvf1A54FrmmpExGvSNoIjAPOBz6aR8wvArelsTwqaTNQBbzeWuX+fStY2c4b2T3B1VdfzeTJk9mwYQPV1dVcf/31xQ7JzMzMSlTx52m3NhG4MSLOaSmQ9ABwJMmUdr7qSZ7BvCSrneck7ZtTbxYwKCKa8xx4/CVJ4puR9EGShPWNTsTV49TW1rJkyZJih2FmZmZloKctkF4P3J5TditwWifbmdRKO7en5VtExMMR8ctOtPtjoFrSEyTT9GdkjaSamZmZ9WpFG7GMiMpWyupaKbsq53hKHm3v10rZl7MOM62cn51z3ADU5JRtAD7T0fXNzMzMeqOeNmJpZmZmZiWqpz1j2SnpTjy5yw69GxGHFyMeMzMzs96spBPLiFgB1HZY0czMzMy6nKfCe7m33nqLiRMnMmLECEaOHMkjjzxS7JDMzMysRJX0iKVtv/PPP5/x48ezYMECNmzYwLp164odkpmZmZWoskosJTUDK0ju60mS5YDWpecmkOzAMzIinpK0I7AMOCWdUkcCdPFcAAAgAElEQVTSDKA6Is5t5xq7pG3fHhGfby+epo3NDJu5sAB3tn0a2lik/e233+bBBx/khhtuAKBfv37069evGyMzMzOzclJuU+FNEVEbETXABiA7QawHHiJdyzIi1pPsPX6tEkOAc4CvdnCN/wAeKHjkRfDss88ycOBAzjzzTMaMGcPZZ5/N2rVrix2WmZmZlSiV0/rekhpb1seUdC4wOiLOk1QJrASOAe6IiBFZ3/kFsBA4Efh1ROS+ZZ7d/mEke4QvAj7U2oilpGnANICqqoGHzbriuoLd37YaNWTXVstXrlzJeeedx9VXX81BBx3E1VdfzYABA5g6dWo3R9gzNTY2Uln5nuVWrUS4/0qX+660uf9K1zHHHPNYRHxoe9ooq6nwFpL6AMeTJIAAJwOLIuJpSaslHRoRS9Nz04FHgVUdJJU7AJcC/wKMbateRMwF5gIMrR4el64o/k/cMLmu1fIRI0ZwySWXcN555wFQUVHBnDlzqKtrvX5vk8lk/FuUMPdf6XLflTb3X+/W6alwSe+XNLorgimA/pKWAUuA54F5aXk9yRaMpP/Wt3whIl4C7gN+0EHb5wF3RcQLBY24iD7wgQ+wzz77sHLlSgDuvfdeDjrooCJHZWZmZqUqr+E0SRngk2n9ZcDrkh7I2SaxJ2iKiK3WtZS0B3AsUCMpgAogJM3I2ud7c/rXno8AR0k6D6gE+qVT7zPb+kL/vhWsbOPFmZ7i6quvZvLkyWzYsIHq6mquv/76YodkZmZmJSrfedpdI+JtSWcD10fENyT9sSsDK6CJwI0RcU5LgaQHgCOBxfk2EhGTs74/heQZyzaTylJRW1vLkiVLih2GmZmZlYF8p8L7SBoMfBq4swvj6Qr1wO05ZbcCpxUhFjMzM7Oyle+I5b8DvwF+GxF/kFQNrOq6sLZNyxvhOWV1rZRdlXM8pZPXuQG4oVPBmZmZmZW5vBLLiLgFuCXr+FngU10VlJmZmZmVnrymwiV9UNK9kp5Ij0dL+nrXhlYckkZJWpbz9/tix2VmZmbW0+X7jOV1JDvSbASIiD+S7mBTbiJiRbp7T/bf4cWOy8zMzKynyzex3CkiHs0p21ToYKz7vfXWW0ycOJERI0YwcuRIHnnkkWKHZGZmZiUq35d33pC0PxAAkiYCL3dZVNZtzj//fMaPH8+CBQvYsGED69atK3ZIZmZmVqLyTSw/R7JN4QhJfwWeAya3/5XuJ6kZWEFyX08CZ0TEuvTcBOA2YGREPCVpR5LF3k+JiBVpnRlAdUSc20rb+6bfrwD6AldHxH+1F0/TxmaGzVxYsPvbVg1tLNL+9ttv8+CDD3LDDTcA0K9fP/r169eNkZmZmVk56XAqPN0j+0MR8X+AgcCIiDgyIv7S5dF1XlP6TGQNsAHIThDrgYdInw2NiPUk+4Rfq8QQ4BySZ0lb8zLw0XRnn8OBmZL26qL76BbPPvssAwcO5Mwzz2TMmDGcffbZrF27tthhmZmZWYnqMLGMiM3A59PPayPinS6PqjAWA8MBJFUCHwPOIuulo4hYRJIwng5cDsyOiDWtNRYRGyLi3fTwfWzDPus9zaZNm1i6dCmf/exnefzxxxkwYABz5swpdlhmZmZWovKdCr9H0gXAz4EtQ1oRsbpLotpOkvoAxwOL0qKTgUUR8bSk1ZIOjYil6bnpwKPAqoj4aQft7gMsJElYvxIRL7VSZxowDaCqaiCzRhX/HadMJtNq+erVq6mqqqKpqYlMJsP+++/Pz372M8aOHdu9AfZQjY2Nbf521vO5/0qX+660uf96t3wTy6npv5/LKgugurDhbLf+kpalnxcD89LP9cAV6ef56fFSgIh4SdJ95LFVZUS8AIxOp8B/KWlBRLyaU2cuyfOoDK0eHpeuyPcn7joNk+vaPHf55ZczePBgDjzwQDKZDEcddRR1dW3X700ymYx/ixLm/itd7rvS5v7r3fLdeWe/rg6kQJrSZyC3kLQHcCxQIylIXr4JSTMiItJqm9O/vKTJ6J+Ao4AFbdXr37eClW28ONNTXH311UyePJkNGzZQXV3N9ddfX+yQzMzMrETllVhKOr218oi4sbDhdImJwI0RcU5LgaQHgCNJRjXzImlv4M2IaJL0fpJnNi8rdLDdrba2liVLlhQ7DDMzMysD+c7T/kPW5x2BsSRTyaWQWNYDuW+k3AqcRicSS2AkcGk66ingey3LFJmZmZlZ/lPhX8g+lrQr0O6LLsUQEZWtlNW1UnZVzvGUPNq+Bxi9HeGZmZmZlbVtXTJnHXBAIQMxMzMzs9KW7zOWvybdzpEkGT0IuKWrgiomSaN472jsuxFxeDHiMTMzMysV+T5j+b2sz5uAv0TEi10QT9Glz03WdljRzMzMzLaS71T4CRHxQPr324h4UdK3uzQy6xZvvfUWEydOZMSIEYwcOZJHHnmk2CGZmZlZico3sRzXStnxhQzEiuP8889n/PjxPPXUUyxfvpyRI0cWOyQzMzMrUe1OhUv6LHAeUC3pj1mndgZ+25WBbQtJzcAKkvt6EjgjItal5yYAtwEjI+IpSTsCy4BTWpYNkjQDqI6Ic9to/wzg6+nhNyPiJ+3F07SxmWEzFxbgzrZPQxuLtL/99ts8+OCD3HDDDQD069ePfv36dWNkZmZmVk46GrH8GfAJ4I7035a/wyLiM10c27ZoiojaiKgBNgDZCWI98BAwCSAi1pPsE36tEkOAc4CvttawpN2BbwCHAx8GvpEulF6ynn32WQYOHMiZZ57JmDFjOPvss1m7dm3HXzQzMzNrRbuJZUT8LSIaIqI+Iv4CNJG8HV4paWi3RLjtFgPDASRVkuyUcxZpYgkQEYuAl4HTgcuB2RGxpo32Pg7cExGr0zr3AOO7Lvyut2nTJpYuXcpnP/tZHn/8cQYMGMCcOblryZuZmZnlJ9/lhj5Bsn3hXsBrwL4kU80Hd11o205SH5JnQBelRScDiyLiaUmrJR0aEUvTc9OBR4FVEdHeou9DgBeyjl9My3KvPQ2YBlBVNZBZozZt380UQCaTabV89erVVFVV0dTURCaTYf/99+dnP/sZY8eO7d4Ae6jGxsY2fzvr+dx/pct9V9rcf71bvssNfRM4AvjfiBgj6RiSqeWepr+kZennxcC89HM9cEX6eX56vBQgIl6SdB9wZwdtq5WyeE9BxFxgLsDQ6uFx6Yp8f+Ku0zC5rs1zl19+OYMHD+bAAw8kk8lw1FFHUVfXdv3eJJPJ+LcoYe6/0uW+K23uv94t36xnY0S8KWkHSTtExP09dLmhpojYag1KSXsAxwI16T7fFUBImhERLYnh5vSvPS8CdVnHewOZQgRdTFdffTWTJ09mw4YNVFdXc/311xc7JDMzMytR+SaWb6XPKS4G/lvSayQLpZeCicCNEXFOS4GkB4AjSe4nX78B/jPrhZ3jaONFnxb9+1awso03snuK2tpalixZUuwwzMzMrAzku47lP5HsDz6d5LnFZ0jeDi8F9cDtOWW3Aqd1ppGIWA38B/CH9O/f0zIzMzMzI88Ry4hYK2lf4ICI+ImknUimlHuUiKhspayulbKrco6n5Nn+j4Efb2N4ZmZmZmUtrxFLSf8KLAB+mBYNAX7ZVUGZmZmZWenJ9xnLz5EsCv57gIhYJWlQl0VVRJJGAbnLDr0bEYcXIx4zMzOzUpFvYvluRGyQkhV30nUi37PUTjlIt3es7bCimZmZmW0l35d3HpD0NZJ1IscBtwC/7rqwzMzMzKzU5DtiOZNkO8QVJPtp3wX8qKuCsq4xbNgwdt55ZyoqKujTp4+XGTIzM7OCajexlDQ0Ip6PiM3AdelfQUhqJklU+5BsD3lGRKxLz00AbgNGRsRTknYElgGnpFPVSJoBVEfEue1c40vAJcCeEfG3tKwOuB84OyLmpWVjSHbi+QqwH8m+4v3SzyvT5r4J1AD/Cryeln0tIu5q6/pNG5sZNnNhZ36W7dKQx5qZ999/P1VVVd0QjZmZmfU2HU2Fb3nzW9KtBb52U0TURkQNsAHIThDrgYeASQARsZ5kDc1rlRhCMnLa7gLlaTt/ACbklK8ATs06ngQsT6/1uXT3nhOAZ9IYayNiQVr38qyyNpNKMzMzs96mo8Qye3/s6i6MYzEwHCDd4edjJFPvk1oqRMQi4GXgdOByYHZErGmrQUn7A5XA13nvvubPAztK2lPJG0njgf8p2N30UJI47rjjOOyww5g7d26xwzEzM7My09EzltHG54JJ3zA/nmRHH4CTgUUR8bSk1ZIOjYil6bnpwKPAqojIXRIoVz1wM0nSeqCkQRHxWtb5BcApwOMk0+Dv5hny5yWdDiwB/m9ucitpGjANoKpqILNGdd/Ol5lMpt3z3/3ud6mqqmLNmjVccMEFNDU1ccghh3RPcCWosbGxw9/Uei73X+ly35U291/v1lFieYikt0lGLvunn0mPIyJ22Y5r95e0LP28GJiXfq4Hrkg/z0+Pl5Jc8CVJ9wF35tH+JGBCRGyWdBtJEnlN1vlfAD8HRpAkoB/No80fkGzrGOm/lwJTsytExFxgLsDQ6uFx6Yp834/afg2T6/Kuu3z5cjZu3EhdXf7f6W0ymYx/nxLm/itd7rvS5v7r3drNeiKiK7dtbEqfZdxC0h7AsUCNpCDZNjIkzYiIlhHTzelfmySNBg4A7knX3uwHPEtWYhkRr0jaCIwDziePxDIiXs26xnXkl+D2CGvXrmXz5s3svPPOrF27lrvvvptZs2YVOywzMzMrI903nJaficCNEXFOS4GkB4AjSUY181VP8gzmJVntPJfud55tFjAoIppbFn9vj6TBEfFyejgBeKK9+v37VrAyjze1u8Orr77KhAnJO0ybNm3itNNOY/z48UWOyszMzMpJT0ss64E5OWW3AqfRucRyEslzm9luT8t/31IQEQ93Mr7vSKolmQpvIHkzvSRUV1ezfPnyYodhZmZmZaxoiWVEVLZSVtdK2VU5x1PyaHu/Vsq+nHWYaeX87JzjBpJ1K7PL/qWja5uZmZn1Vvlu6WhmZmZm1q6eNhXeKZJGAbnLDr0bEYcXIx4zMzOz3qykE8t0e8faDiuamZmZWZfzVLiZmZmZFURJj1ha5wwbNoydd96ZiooK+vTpw5IlS4odkpmZmZURJ5ZdqGljM8NmLuy26zXksWbm/fffT1VVVTdEY2ZmZr1NWU2FS2qWtEzSE5JukbRT1rkJkkLSiPR4R0lPpS8AtdSZIem/2mi7VtIjkv4k6Y+STu36OzIzMzMrHWWVWJJuExkRNcAG4Nysc/XAQySLpBMR64HpwLVKDCFZ8PyrbbS9Djg9Ig4GxgNXSNqti+6jS0jiuOOO47DDDmPu3LnFDsfMzMzKjP6+BXfpk9TYsvC6pHOB0RFxnqRKYCVwDHBHRIzI+s4vgIXAicCvIyJ3+aK2rrUcmBgRq3LKpwHTAKqqBh4264rrCnBn+Rk1ZNd2z7/xxhtUVVWxZs0aLrjgAr74xS9yyCGHdFN0paexsZHKyves428lwv1Xutx3pc39V7qOOeaYxyLiQ9vTRlk+YympD8mWjovSopOBRRHxtKTVkg6NiKXpuenAo8CqTiSVHwb6Ac/knouIucBcgKHVw+PSFd33EzdMrsu77vLly9m4cSN1dfl/p7fJZDL+fUqY+690ue9Km/uvdyu3qfD+kpYBS4DngXlpeT0wP/08Pz0GICJeAu4DfpDPBSQNJlmU/cyI2FyguLvc2rVreeedd7Z8vvvuu6mpqengW2ZmZmb5K7cRy6aI2GrBdEl7AMcCNZICqABC0oz4+3MAm9O/dknahWTa/OsR8buO6vfvW8HKPN7U7g6vvvoqEyZMAGDTpk2cdtppjB8/vshRmZmZWTkpt8SyNROBGyPinJYCSQ8ARwKL821EUj/g9rStWwoeZRerrq5m+fLlxQ7DzMzMyli5TYW3pp4kIcx2K3BaJ9v5NHA0MCVd0miZJG8naWZmZpYqqxHLljfCc8rqWim7Kud4Sh5t3wTctB3hmZmZmZW13jBiaWZmZmbdoKxGLAsh3Yknd9mhdyPi8GLEY2ZmZlYqnFjmiIgVgJ+dNDMzM+skT4WXqebmZsaMGcNJJ51U7FDMzMysl3BiWaauvPJKRo4cWewwzMzMrBcp2lS4pGZgRRrDk8AZEbEuPTcBuA0YGRFPSdoRWAackk5VI2kGUB0R57ZzjS8BlwB7RsTf0rI64H7g7IiYl5aNAZYCXwH2Az5GsmXjfiR7jAN8ExAwGxgJfDgilrR3j00bmxk2c2EnfpX8NbSz8PqLL77IwoULueiii7jsssu65PpmZmZmuYo5YtkUEbURUQNsALITxHrgIWASQESsJ9nT+1olhgDnAF/t4Br1wB+ACTnlK4BTs44nAcvTa30u3b3nBOCZNMbaiFgAPAH8M/Bgp++2G02fPp3vfOc77LCDB6TNzMys+/SUl3cWA6MBJFWSjBgeA9xBMkJIRCySNBU4HTgRmB0Ra9pqUNL+QCXJKOTXgBuyTj8P7CJpT+A1YDxwV0dBRsSTadtt1pE0DZgGUFU1kFmjNnXU7DbJZDKtlj/yyCNs3LiRd955h2XLlvHmm2+2Wdfa19jY6N+uhLn/Spf7rrS5/3q3oieWkvoAxwOL0qKTgUUR8bSk1ZIOjYil6bnpwKPAqojIXRIoVz1wM0nSeqCkQRHxWtb5BcApwOMk0+DvFuJ+ImIuMBdgaPXwuHRF1/zEDZPrWi3/zW9+w2OPPcaUKVNYv349b7/9Nj/60Y+46Sav7d5ZmUyGurq6Yodh28j9V7rcd6XN/de7FXOutL+kZcASkhHEeWl5PTA//Tw/PQYgIl4C7gN+kEf7k4D5EbGZ5HnNU3LO/yIta0lAy8Ill1zCiy++SENDA/Pnz+fYY491UmlmZmbdopgjlk3ps4xbSNoDOBaokRRABRCSZkREpNU2p39tkjQaOAC4J5227gc8C1zTUiciXpG0ERgHnA98tCB3laV/3wpWtvOSjZmZmVk56Wlvd0wEboyIfSNiWETsAzwHHNnJdupJnsEclv7tBQyRtG9OvVnAhRHRvP2h9zx1dXXceeedxQ7DzMzMeomelljWA7fnlN0KnNbJdia10s7tafkWEfFwRPwy30YlTZD0IvARYKGk33QyLjMzM7OyVbSp8IiobKWsrpWyq3KOp+TR9n6tlH056zDTyvnZOccNQE1O2e28N2E1MzMzM3reiKWZmZmZlaiiLze0PSSNAnKXHXo3Ig4vRjxmZmZmvVlJJ5bp9o61HVY0MzMzsy7nqfAy1dzczJgxYzjppJOKHYqZmZn1Ek4sy9SVV17JyJEjix2GmZmZ9SIlPRWeS1IzsILkvp4EzoiIdem5CSQ78IyMiKck7QgsA05Jp9SRNAOojohzO2gf4PmI+GR78TRtbGbYzIUFuLP3amhn4fUXX3yRhQsXctFFF3HZZZd1yfXNzMzMcpXbiGVTRNRGRA2wAchOEOuBh0jXsoyI9SR7j1+rxBDgHOCrebRf21FSWUzTp0/nO9/5DjvsUG7da2ZmZj1ZOWcei4HhAJIqgY8BZ5G1SHpELAJeBk4HLifZrWdN94daOHfeeSeDBg3isMMOK3YoZmZm1svo71twlz5JjRFRKakPyY49iyLiB5I+AxwTEWdJehj4fEQsTb+zF/AosCoijumg/U0k0+ebgDmt7dojaRowDaCqauBhs664rpC3uMWoIbu2Wn7ddddx9913U1FRwYYNG1i3bh1HHXUUF110UZfEUc4aGxuprHzPOv5WItx/pct9V9rcf6XrmGOOeSwiPrQ9bZTVM5ZAf0nL0s+LgXnp53rgivTz/PR4KUBEvCTpPiCfTbWHpvWrgfskrYiIZ7IrRMRcYC7A0OrhcemKrvmJGybXtVpeV/f38kwmw/e+9z3vF76NMpnMVr+nlRb3X+ly35U291/vVm6JZVNEbLWupaQ9gGOBGkkBVAAhaUb8fbh2c/rXroh4Kf33WUkZYAzwTFv1+/etYGU7L9mYmZmZlZNyfsayxUTgxojYNyKGRcQ+wHPAkZ1pRNL7Jb0v/VxF8szmnwsebQHV1dV5tNLMzMy6TW9ILOuB23PKbgVO62Q7I4ElkpYD95M8Y9mjE0szMzOz7lRWU+ER8Z6nhSOirpWyq3KOp+TR9sPAqO0Iz8zMzKys9YYRSzMzMzPrBmU1YlkIkkYBP80pfjciDi9GPGZmZmalwolljnR7x9oOK5qZmZnZVjwVXsLWr1/Phz/8YQ455BAOPvhgvvGNbxQ7JDMzM+vFPGJZwt73vvdx3333UVlZycaNGznyyCM5/vjjOeKII4odmpmZmfVCRUssJTUDK9IYngTOiIh16bkJwG3AyIh4StKOJFspnpJOVSNpBlAdEee2c40vAZcAe0bE39KyOpLlgs6OiHlp2RiSnXi+AuxHskZlv/TzyrS5b0bEgrT+BcB3gYER8UZb12/a2MywmQs7+9NspaGdBdYlbdk2a+PGjWzcuBFJ23U9MzMzs21VzKnwpoiojYgaYAOQnSDWAw8BkwAiYj0wHbhWiSHAOcBXO7hGPfAHYEJO+Qrg1KzjScDy9FqfS3fvOQF4Jo2xNiup3AcYBzzf2RvuCs3NzdTW1jJo0CDGjRvH4Yf7HSMzMzMrjp7yjOViYDiApEqSEcOzSBNLgIhYBLwMnA5cDsyOiDVtNShpf6AS+DpJgpnteWBHSXsqGeIbD/xPnrFeDswAoqOK3aGiooJly5bx4osv8uijj/LEE08UOyQzMzPrpYr+jKWkPsDxwKK06GRgUUQ8LWm1pEMjYml6bjrwKLAqInKXBMpVD9xMkrQeKGlQRLyWdX4BcArwOMk0+Lt5xPpJ4K8RsbytKWdJ04BpAFVVA5k1alNHzbYrk8nkXXfYsGFcc801nHrqqR1Xtg41NjZ26ve3nsX9V7rcd6XN/de7FTOx7C9pWfp5MTAv/VwPXJF+np8eLwWIiJck3QfkswH2JGBCRGyWdBtJEnlN1vlfAD8HRpAkoB9trzFJOwEXAce1Vy8i5gJzAYZWD49LV2zfT9wwua7Nc6+//jp9+/Zlt912o6mpiX/7t3/jwgsvpK6u7e9Y/jKZjH/LEub+K13uu9Lm/uvdiplYNqXPMm4haQ/gWKBGUgAVQEiaEREtU8+b0782SRrN/2/v3sOsqs48j39/logIXuIA3cRLl7cOyK1iGDAOIUUMogxGyYhKmA4oDJqME9tLEsw8aptuMXaGnpDEOE0narR90HTihUhHEjXHRkWNchGi0dihOuBgYhxvBYUpi3f+2Kvs47Eup6pOsetQv8/z8NTZa6+91nvOerZ5sy9rwXHAz9KVxf2A31CUWEbEy5KayZ6XvJhOEkvgGLKXeVqvVh4OrJM0MSJeLuP7Vtz27duZN28eLS0t7N69m7PPPpuZM2fmEYqZmZlZ/rfCS5wF3BoRF7QWSHoYmEx2VbNcc8iewbyuqJ0tkv6spN5VwPCIaOnsber0NvrwovYagAkdvRU+aEANz3fwVndPjRs3jvXr1/da+2ZmZmZd0Vde3mk1B7i7pOxHwGe62M65bbRzN0UvAwFExGMRcU8X2zYzMzOzNuR2xTIihrRRVt9G2TdLtueX0fZRbZRdWrRZaGP/X5VsNwBjOuijtrM4zMzMzPqTvnbF0szMzMyqVF97xrJLJI0FSqcdejsiPEu4mZmZ2R5W1YlleqGmrtOKZmZmZtbrfCu8iu3atYuJEycyfvx4Ro8ezdVXX513SGZmZtaPVfUVy/5u4MCBPPTQQwwZMoTm5mYmT57Maaedxoknnph3aGZmZtYP7fHEUlILsAkQ0AJcFBGPSaoF7ouIMZLqgcsjYmbJsYVU/lSZfS0jmxvziIjYncrmAzcDn4yIB1PZLKB1dZ65ZBOhDwGGAVtSc5+PiMdS/W8B57X1ZnuxpuYWahevKifUdjV0MA+mJIYMyUJobm6mubmZzubjNDMzM+stedwKb4qIuogYD1wBXNfZAd0haR9gFrAVmFKyexPZnJmtzgU2AkTErLQi0EJgTYq1riipnAAc0hsxd0dLSwt1dXUMHz6cadOmMWmS31syMzOzfOT9jOVBwGu91PZUYDNwI+9NIiFbxWeipAGShgDHAhvohKQa4OvAlyoca7fV1NSwYcMGtm3bxpNPPsnmzZvzDsnMzMz6qTyesRwkaQOwPzCCbG3w3jAHWAHcCyyRNCAimtO+AB4ApgMHAyvJbn935iJgZURsb++Ws6RFwCKAoUOHcdXYd3r0JQqFQtl1a2trueGGGzjnnHN61KdlGhsbu/T7W9/i8ateHrvq5vHr3/JILJvSrWYkfRS4VVK7K9x0h6T9gBnAJRHxlqQngFOA4gce7wC+QJZYXgZ8pZM2P0j2DGZ9R/UiYjmwHODIo4+NpZt69hM3zG2/u1deeYUBAwZwyCGH0NTUxJVXXsmXv/xl6us7DNHKVCgU/FtWMY9f9fLYVTePX/+W61vhEbFW0lCyl2Qq6VSyhHFTurJ4ALCTosQyIp5MCW1TRLxQxksvHya7Zf5ia5uSXoyIYysce9m2b9/OvHnzaGlpYffu3Zx99tnMnDmz8wPNzMzMekGuiaWkkUAN8CpZ8lcpc4CFEbEi9TMY2CKptI8rgF3lNBgRq4A/bd2W1NhZUjloQA3Pd/BWd0+NGzeO9evX91r7ZmZmZl2R5zOWkE05NC8iWtq4YniypG1F27PT31WSWp+VXBsRs4sPSsnjdOCC1rKI2CHpEeD04roR8ZOefRUzMzMza7XHE8uIqGmnvAEYkz4XgEFtVKsvo/2dwKFtlH+6aPOWNvbPL9kuAIUO+ulwDkszMzOz/ibv6YbMzMzMbC9R1Us6SpoOXF9SvCUiZuURj5mZmVl/VtWJZUSsBlbnHYeZmZmZ+Va4mZmZmVWIE8sqtmvXLiZOnMj48eMZPXo0V199dd4hmZmZWT9W1e511hEAABg1SURBVLfC+7qm5hZqF6/qvGIHGjqYB3PgwIE89NBDDBkyhObmZiZPnsxpp53GiSee2KM+zczMzLpjj1+xlNQiaYOkjZLWSToplddK2pw+10u6r41jC5ImdKGvZZJekrRPUdl8SSHp5KKyWansLEl3p/helPRG+rxB0kmSbpf0vKTNkm6SNKBnv0bPSGLIkGzWo+bmZpqbmyljBSEzMzOzXpHHrfCmiKiLiPFkK99c1xudpGRyFrAVmFKyexPZ6jytzgU2AkTErLSW+UJgTYq1LiIeA24HRgJjyebZXNgbsXdFS0sLdXV1DB8+nGnTpjFp0qS8QzIzM7N+Ku9b4QcBr/VS21OBzcCdZElkoWjfGuBj6YrjQLI1wDeUNlAqIv659bOkJ4HDS+tIWgQsAhg6dBhXjX2n+98AKBQKndb5xje+QWNjI1deeSUjR47kqKOO6lGflmlsbCzr97e+yeNXvTx21c3j17/luaTj/sAI4BO91M8cYAVwL7BE0oCIaF0KMoAHyJZ+PBhYCZSdjaWE9C+Ai0v3RcRyYDnAkUcfG0s39ewnbphbX3bdp59+mldffZXzzjuvR31aplAoUF9fn3cY1k0ev+rlsatuHr/+Lc9b4SOBU4FbVeEHAyXtB8wA7omIN4EngFNKqt1Bdgv8XLIEtCu+A/xLRKzpaaw98corr/D6668D0NTUxAMPPMDIkSPzDMnMzMz6sVxvhUfEWklDgWEVbvpUsiuRm1LOegCwE3j3Fe2IeFLSGLJE94Vyc1tJV6d4L+is7qABNTzfwVvdPbV9+3bmzZtHS0sLu3fv5uyzz2bmzJm91p+ZmZlZR3JNLCWNBGqAV8mSv0qZAyyMiBWpn8HAFkmlfVwB7Cq3UUkLyW6fnxwRuysVbHeNGzeO9evX5x2GmZmZGZDvM5YAAuZFREsbVwxPlrStaHt2+rtKUuuzkmsjYnbxQSl5nE7RFcWI2CHpEeD04roR8ZMuxv5/gH8D1qZ474qIr3axDTMzM7O90h5PLCOipp3yBmBM+lwgm86nVH0Z7e8EDm2j/NNFm7e0sX9+yXaB975JTkTk/Ra9mZmZWZ/lJR3NzMzMrCKq+gqcpOnA9SXFWyJiVh7xmJmZmfVnVZ1YRsRqYHXecZiZmZmZb4VXha1btzJ16lRGjRrF6NGjWbZsWd4hmZmZmb1PVV+x7OuamluoXbyq84pJQztzXu67774sXbqUE044gbfeeouPfOQjTJs2jeOPP75SoZqZmZn12B6/YimpRdIGSRslrZN0UiqvlbQ5fa6XdF8bxxYkTehCX8skvSRpn6Ky+ZJC0slFZbNS2VmS7k7xvSjpjfR5g6STlLlW0guSnpP0hZ79GuUZMWIEJ5xwAgAHHnggo0aN4qWXXtoTXZuZmZmVLY8rlk0RUQfvvnxzHfDxSneSkslZwFZgCu+dOmgT2STqD6btc4GNAK0v/kiqBy6PiHeXspF0HnAEMDIidksaXum4O9PQ0MD69euZNGnSnu7azMzMrEN53wo/CHitl9qeCmwG7iRLIgtF+9YAH5M0ABgIHAtsKG2gDZ8DPtO66k5E/L60gqRFwCKAoUOHcdXYd8oOuFAodLi/qamJiy++mIULF7Ju3bqy27XuaWxs7HRMrO/y+FUvj1118/j1b3muvLM/MAL4RC/1MwdYAdwLLJE0ICJaV+wJ4AGyFXoOBlYCR5XR5jHAOZJmAa8AX4iIXxdXiIjlwHKAI48+NpZuKv8nbphb3+6+5uZmZs6cyYUXXsill15adpvWfYVCgfr6+rzDsG7y+FUvj1118/j1b3m8Fd4UEXURMRI4FbhVbazn2BOS9gNmAPdExJvAE8ApJdXuILsFfi5ZAlqOgcCuiJgA/ANwU2Ui7lhEsGDBAkaNGuWk0szMzPqsXG+FR8RaSUOBYRVu+lSyK5GbUs56ALATePcV7Yh4UtIYskT3hTJz223Aj9Lnu4GbO6o8aEANz7fzpndXPProo9x2222MHTuWuro6AJYsWcKMGTN63LaZmZlZpeSaWEoaCdQAr5Ilf5UyB1gYEStSP4OBLZJK+7gC2NWFdu8hu3V/E9kLRy9UINZOTZ48mYjYE12ZmZmZdVuez1gCCJgXES1tXDE8WdK2ou3Z6e8qSa3PSq6NiNnFB6XkcTpwQWtZROyQ9AhwenHdiPhJF2P/GnC7pEuARmBhF483MzMz22vt8cQyImraKW8AxqTPBWBQG9Xqy2h/J3BoG+WfLtq8pY3980u2C7z3TXIi4nWg5/e2zczMzPZCXtLRzMzMzCoi73kseyRNsH59SfGW1knOzczMzGzPqerEMiJWA6vzjsPMzMzMfCu8KmzdupWpU6cyatQoRo8ezbJly/IOyczMzOx9ei2xlNQiaUPRv8WpvCDpeUnPSPqVpG9LOiTtq5W0uaSdv5J0edH25em4zZI2Svps0b5hkpolXVBU9kTq/7eSXimKp1ZSQ5pHE0mHS7pX0q8l/aukZWmidSTVSwpJpxe1e19aT7zX7bvvvixdupTnnnuOxx9/nBtuuIFnn312T3RtZmZmVrbevBXeFBF17eybGxFPpcTtOrJlFz/eWYOSLgSmARMj4k1JBwNnFlWZDTxONo/l3wNExKR07HxgQkRcVNRe618BdwE3RsQZkmrIlmW8Fvhiqr4N+J/Ajzv/6pmm5hZqF6/qvGLS0M5k6iNGjGDEiBEAHHjggYwaNYqXXnqJ448/vuy2zczMzHpbrrfCI+KPwJeAIyWNL+OQrwCfT8s0EhFvRMT3i/bPAS4DDpd0WBdC+QTZUo03p3ZbgEuA84smVd8IvCFpWhfarbiGhgbWr1/PpEmT8gzDzMzM7H1684pl8UToANdFxJ2lldLk6BuBkWRrerdJ0oHAgRHxr+3sPwL407RU4w+Ac4C/KzPW0cDTJXG9Kem3wLFFxX+T/v2sgzgXAYsAhg4dxlVj3ykzBCgUCh3ub2pq4uKLL2bhwoWsW7eu7HatexobGzsdE+u7PH7Vy2NX3Tx+/Vtet8JLtS670966hZHqdLSu4bnAD9LnO4DvUX5i2V7b7ymPiDWSkPSx9hqKiOVkt9E58uhjY+mm8n/ihrn17e5rbm5m5syZXHjhhVx66aVlt2ndVygUqK+vzzsM6yaPX/Xy2FU3j1//lvtb4el5xrHAc2Rrhn+gpMqhwB/S7e8dko5up6k5wHxJDcBKYLyk48oM45fAhJK4DgKOAEqvkF5L9qzlHhMRLFiwgFGjRjmpNDMzsz4r13ksJQ0gS9S2RsQzqWy7pJMj4kFJhwKnAq3z61wH3CDpnHSr+iCyK5UPA4Mj4rCitq9J+/66jFAeBL4m6bMRcWtKdpcCt0TEzuJ1zCPip5L+GvhgZ40OGlDD8+28kNMVjz76KLfddhtjx46lri67CLxkyRJmzJjR47bNzMzMKmVPPmN5f0QsTp9vl/Q2MBB4ADijqN5nyZLHpWn7mqLnKm8EhgC/kNQMNJMlgHOAu0v6/xHZLfFOE8uICEmzgO9IupLsSu4/k70s1JZryd5k3yMmT55MREdPAZiZmZnlr9cSy4ioaae8vpPjngWmtrMvgL9N/zrr/xng+KLtW4BbSurUFn3eCpxOGyKiABSKtlfy78+FmpmZmRl94BlLMzMzM9s7OLE0MzMzs4pwYmlmZmZmFeHE0szMzMwqwollH3P++eczfPhwxowZk3coZmZmZl3ixLKPmT9/Pvfff3/eYZiZmZl12R6fIF1SC7CJbLqeFuCiiHhMUi1wX0SMkVQPXB4RM0uOLaTyp8rsaxlwFnBEROxOZfOBm4FPRsSDqWwWcBcwG5gLHEU2X+YwYEtq7vNka4B/HHgjlc2PiOK5Ot+jqbmF2sWr3lfe0MGk6VOmTKGhoaGcr2dmZmbWp+RxxbIpIuoiYjxwBdlqOhUnaR9gFrAVmFKyexPZpOqtzgU2AkTErLTG+UJgTYq1LiIeS3W/WFTWblJpZmZm1t/kfSv8IOC1Xmp7KrCZbLWeOSX71gATJQ2QNAQ4FnCSaGZmZtYDeawV3rrU4/7ACOATvdTPHGAF2dKLSyQNiIjmtC/IlpKcDhwMrCS7/V2OayVdRba++OKIeLt4p6RFZLfMGTp0GFeNfed9DRQKhQ47ePnll9mxY0en9ax3NTY2egyqmMevennsqpvHr3/LI7FsSreakfRR4FZJFX0FWtJ+wAzgkoh4S9ITwClA8QOPdwBfIEssL6P9dcGLXQG8DOwHLAe+DHy1uEJELE/7OPLoY2Pppvf/xA1z6zvspKGhgcGDB1Nf33E9612FQsFjUMU8ftXLY1fdPH79Wx6J5bsiYq2koWQvyVTSqWQJ4yZJAAcAOylKLCPiyZTQNkXEC6leZ/FuTx/flnQzcHlH9QcNqOH5Dl7UacucOXMoFAr84Q9/4PDDD+eaa65hwYIFXWrDzMzMLA+5JpaSRgI1wKtkyV+lzAEWRsSK1M9gYIuk0j6uAHaV26ikERGxXVkWeibZM5wVtWLFiko3aWZmZrZH5PmMJWRTDs2LiJY2rhieLGlb0fbs9HeVpNZnJddGxOzig1LyOB24oLUsInZIegQ4vbhuRPyki7HfLmlYinsDcGEXjzczMzPba+3xxDIiatopbwDGpM8FYFAb1erLaH8ncGgb5Z8u2ryljf3zS7YLQKGkrLdeNDIzMzOrenlPN2RmZmZme4lcn7HsKUnTgetLirdExKw84jEzMzPrz6o6sYyI1cDqvOMwMzMzM98K73POP/98hg8fzpgxFZ3a08zMzKzXObHsY+bPn8/999+fdxhmZmZmXbbHb4VLagE2kU3Z0wJcFBGPSaoF7ouIMZLqgcsjYmbJsYVU/lSZfS0DzgKOiIjdqWw+cDPwyYh4MJXNAu4im9JoLtnyjkPIJm7fkpr7PLAAmJBifwGYHxGN7fXf1NxC7eJV7ytv6GDS9ClTptDQ0FDO1zMzMzPrU/K4YtkUEXURMZ5sgvLreqMTSfsAs4CtwJSS3ZvIJlFvdS6wESAiZqUlJxcCa1KsdRHxGNkSkeMjYhzwW+Ci3ojdzMzMrBrlfSv8IOC1Xmp7KtnKODfy3iQSYA0wUdIASUOAY8kmPO9QRLwJkFbeGQRERSM2MzMzq2J5rryzPzAC6K1Jx+cAK4B7gSWSBkRE64o9ATxAtkLPwcBKstvfnUprhM8AngUua2P/ImBR2nyb62e+b9lHlU6Q9H77AcdJ+mU5MVmvGQr8Ie8grNs8ftXLY1fdPH7V60M9bSCPxLIp3WpG0keBWyVV9BVoSfuRJX+XRMRbkp4ATgGKH3i8A/gCWWJ5GfCVctqOiPMk1QDfAs4he16zeP9yYHmK46mImNCN+GvJnjft8rFWOd0dP+sbPH7Vy2NX3Tx+1UtSWe+wdCTXW+ERsZbs/9kMq3DTp5IljJskNQCTKbkdHhFPki0hOTQiXuhK4xHRAtwJ/JeKRFtE0gpgLfAhSdskLah0H2ZmZma9IdcJ0iWNBGqAV4EDKtj0HGBhRKxI/QwGtkgq7eMKYFeZsQo4JiJeTJ9PB35VwZgBiIjS50HNzMzMqkKez1hCNm3PvIhoyXK19zhZ0rai7dnp7ypJrc9Kro2I2cUHpeRxOnBBa1lE7JD0CFkySFH5T7oQt4DvSzoofd4IfK6TY5Z3oX3rezx+1c3jV708dtXN41e9ejx2ivCLzWZmZmbWc3lPN2RmZmZme4mqTiwlTZe0oeTf3XnHBSDpVEnPS3pR0uK847H2STpC0s8lPSfpl5IuTuWHSvqZpF+nvx/IO1Zrn6QaSesl3Ze2j5L0RBq/O9NsEdYHSTpE0g8l/Sqdhx/1+VcdJF2S/ru5WdIKSfv73Ou7JN0k6feSNheVtXmuKfPNlMc8I+mEcvqo6sQyIlYXrYzT+m9W3nGl6YhuAE4DjgfmSDo+36isA+8Al0XEKOBE4L+n8VoMPBgRxwEPpm3ruy4Gnivavh7432n8XiNbktX6pmXA/RExEhhPNo4+//o4SYeRTds3ISLGkL2Mey4+9/qyW8hmzinW3rl2GnBc+reIbMGZTlV1YtmHTQRejIjfRMQfyebMPCPnmKwdEbE9Italz2+R/Y/aYWRj9v1U7fvAmflEaJ2RdDjwn4Hvpm2RLb7ww1TF49dHpRcipwDfA4iIP0bE6/j8qxb7kr2Uuy/Z7C7b8bnXZ0XEvwD/r6S4vXPtDODWyDwOHCJpRGd9OLHsHYeRrVHealsqsz4uTU7/YeAJ4E8iYjtkyScwPL/IrBPfAL4E7E7b/wF4PSLeSds+B/uuo4FXgJvTowzfTVPE+fzr4yLiJeB/Ab8lSyjfAJ7G5161ae9c61Yu48Syd7xv7iS8rnifl9aN/xHwl63rwlvfJ2km8PuIeLq4uI2qPgf7pn2BE4AbI+LDwA5827sqpGfxziBbEvmDwGCy26elfO5Vp279d9SJZe/YBhxRtH048H9zisXKIGkAWVJ5e0TclYp/13rZP/39fV7xWYf+E/CptMrWHWS34b5Bdtumda5en4N91zZgW0Q8kbZ/SJZo+vzr+z4JbImIVyKiGbgLOAmfe9WmvXOtW7mME8ve8QvguPRm3H5kDzOvzDkma0d6Hu97wHMR8XdFu1YC89LnecC9ezo261xEXBERh0dELdm59lBEzAV+DpyVqnn8+qiIeBnYKulDqehk4Fl8/lWD3wInSjog/Xe0dex87lWX9s61lcBn09vhJwJvtN4y74gnSO8lkmaQXTWpAW6KiGtzDsnaIWkysAbYxL8/o/cVsucsfwAcSfYf0NkRUfrQs/UhkuqByyNipqSjya5gHgqsB/5rRLydZ3zWNkl1ZC9e7Qf8BjiP7MKHz78+TtI1wDlks2usBxaSPYfnc68PkrQCqAeGAr8DrgbuoY1zLf2fhW+TvUW+EzgvIp7qtA8nlmZmZmZWCb4VbmZmZmYV4cTSzMzMzCrCiaWZmZmZVYQTSzMzMzOrCCeWZmZmZlYRTizNrN+R1CJpQ9G/2m60cYikz1c+unfb/5SkPboCjaQzJR2/J/s0s72Lpxsys35HUmNEDOlhG7XAfRExpovH1URES0/67g1ppZTvkn2nH+Ydj5lVJ1+xNDMjS/gkfV3SLyQ9I+mCVD5E0oOS1knaJOmMdMjXgGPSFc+vS6qXdF9Re9+WND99bpB0laRHgNmSjpF0v6SnJa2RNLKNeOZL+nb6fIukGyX9XNJvJH1c0k2SnpN0S9ExjZKWplgflDQslddJejx9r7vTGs9IKkhaIulh4MvAp4Cvp+90jKT/ln6PjZJ+JOmAoni+KemxFM9ZRTF8Kf1OGyV9LZV1+n3NbO+wb+dVzMz2OoMkbUift0TELGAB2ZJl/1HSQOBRST8FtgKzIuJNSUOBxyWtBBYDYyKiDt5d9acjuyJicqr7IHBhRPxa0iTgO2RrnHfkA6nOp4Afk62RvhD4haS6iNgADAbWRcRlkq4iW1XjIuBW4H9ExMOSvprK/zK1e0hEfDzFdRxFVywlvR4R/5A+/036jb6VjhsBTAZGki399kNJpwFnApMiYqekQ1Pd5d34vmZWhZxYmll/1NSaEBY5BRhXdPXtYOA4YBuwRNIUsiU/DwP+pBt93gnZFVDgJOCfshXTABhYxvE/joiQtAn4XURsSu39EqgFNqT47kz1/xG4S9LBZMnjw6n8+8A/lcbVjjEpoTwEGAKsLtp3T0TsBp6V1Pp7fBK4OSJ2AqRl4br7fc2sCjmxNDPLiOyq3ur3FGa3s4cBH4mIZkkNwP5tHP8O7328qLTOjvR3H+D1NhLbzrSutby76HPrdnv/LS/nIfodHey7BTgzIjam36G+jXgg++1a/5b22d3va2ZVyM9YmpllVgOfkzQAQNKfSxpMduXy9ympnAr8War/FnBg0fH/BhwvaWC6SnhyW51ExJvAFkmzUz+SNL5C32EfoPWK62eARyLiDeA1SR9L5X8BPNzWwbz/Ox0IbE+/ydwy+v8pcH7Rs5iH9vL3NbM+xomlmVnmu8CzwDpJm4G/J7sSeDswQdJTZMnVrwAi4lWy5zA3S/p6RGwFfgA8k45Z30Ffc4EFkjYCvwTO6KBuV+wARkt6muwZxq+m8nlkL+U8A9QVlZe6A/iipPWSjgGuBJ4Afkb63h2JiPvJnrd8Kj3Denna1Vvf18z6GE83ZGa2l1AFplEyM+sJX7E0MzMzs4rwFUszMzMzqwhfsTQzMzOzinBiaWZmZmYV4cTSzMzMzCrCiaWZmZmZVYQTSzMzMzOrCCeWZmZmZlYR/x+gQiVlFXWaiwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = lgb.plot_importance(gbm, height = 0.4, \n",
    "                         max_num_features = 25, \n",
    "                         xlim = (0,100), ylim = (0,23), \n",
    "                         figsize = (10,6))\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Dimensionality reduction using feature importances</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[15, 11, 10, 10, 9, 8, 6, 6, 6, 6, 5, 5, 4, 4, 3, 3, 3, 2, 2, 1, 1, 0, 0]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For each feature of our dataset, the result of the following\n",
    "# code snippet contains numbers of times a feature is used in a model.\n",
    "sorted(gbm.feature_importances_,reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15\n",
      "11\n",
      "10\n",
      "10\n",
      "9\n",
      "8\n",
      "6\n",
      "6\n",
      "6\n",
      "6\n",
      "5\n",
      "5\n",
      "4\n",
      "4\n",
      "4 0.875\n"
     ]
    }
   ],
   "source": [
    "# The code below aims to drop  to keep the features that are included in the most important features. \n",
    "temp = 0 \n",
    "total = sum(gbm.feature_importances_)\n",
    "for feature in sorted(gbm.feature_importances_, reverse=True):\n",
    "    temp+=feature\n",
    "    print(feature)\n",
    "    if temp/total >= 0.85:\n",
    "        print(feature,temp/total) # stop when we \n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC without dimensionality reduction: \n",
      "0.7686460477376955\n"
     ]
    }
   ],
   "source": [
    "#The above means let go of all variables after PAY_AMT_5\n",
    "y_pred_prob = gbm.predict_proba(X_test)[:, 1]\n",
    "auc_roc_0 = str(roc_auc_score(y_test, y_pred_prob)) # store AUC score without dimensionality reduction\n",
    "print('AUC without dimensionality reduction: \\n' + auc_roc_0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We can choose to drop the last 6 features from in our new model to reduce dimensionality, and thus save training time and space\n",
    "\n",
    "X = X.drop(['PAY_5','PAY_AMT4','BILL_AMT4','BILL_AMT2','BILL_AMT6','EDUCATION','BILL_AMT5','BILL_AMT3'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ajc/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_split.py:2053: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.649987\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614878\tvalid_0's auc: 0.765334\n",
      "[3]\tvalid_0's binary_logloss: 0.585872\tvalid_0's auc: 0.769379\n",
      "[4]\tvalid_0's binary_logloss: 0.561615\tvalid_0's auc: 0.770104\n",
      "[5]\tvalid_0's binary_logloss: 0.541663\tvalid_0's auc: 0.77207\n",
      "[6]\tvalid_0's binary_logloss: 0.525011\tvalid_0's auc: 0.771646\n",
      "[7]\tvalid_0's binary_logloss: 0.511302\tvalid_0's auc: 0.770342\n",
      "[8]\tvalid_0's binary_logloss: 0.499137\tvalid_0's auc: 0.772387\n",
      "[9]\tvalid_0's binary_logloss: 0.488921\tvalid_0's auc: 0.772538\n",
      "[10]\tvalid_0's binary_logloss: 0.480526\tvalid_0's auc: 0.77215\n",
      "[11]\tvalid_0's binary_logloss: 0.473288\tvalid_0's auc: 0.772052\n",
      "[12]\tvalid_0's binary_logloss: 0.467178\tvalid_0's auc: 0.772153\n",
      "[13]\tvalid_0's binary_logloss: 0.461907\tvalid_0's auc: 0.772628\n",
      "[14]\tvalid_0's binary_logloss: 0.457719\tvalid_0's auc: 0.772073\n",
      "[15]\tvalid_0's binary_logloss: 0.454001\tvalid_0's auc: 0.772137\n",
      "[16]\tvalid_0's binary_logloss: 0.450598\tvalid_0's auc: 0.772892\n",
      "[17]\tvalid_0's binary_logloss: 0.447852\tvalid_0's auc: 0.773144\n",
      "[18]\tvalid_0's binary_logloss: 0.44543\tvalid_0's auc: 0.773478\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "[1]\tvalid_0's binary_logloss: 0.650046\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614805\tvalid_0's auc: 0.768218\n",
      "[3]\tvalid_0's binary_logloss: 0.585989\tvalid_0's auc: 0.766994\n",
      "[4]\tvalid_0's binary_logloss: 0.56196\tvalid_0's auc: 0.767616\n",
      "[5]\tvalid_0's binary_logloss: 0.541971\tvalid_0's auc: 0.768534\n",
      "[6]\tvalid_0's binary_logloss: 0.525079\tvalid_0's auc: 0.769288\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[8]\tvalid_0's binary_logloss: 0.499175\tvalid_0's auc: 0.770306\n",
      "[9]\tvalid_0's binary_logloss: 0.489026\tvalid_0's auc: 0.770181\n",
      "[10]\tvalid_0's binary_logloss: 0.480527\tvalid_0's auc: 0.770099\n",
      "[11]\tvalid_0's binary_logloss: 0.47319\tvalid_0's auc: 0.770482\n",
      "[12]\tvalid_0's binary_logloss: 0.46736\tvalid_0's auc: 0.770093\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[1]\tvalid_0's binary_logloss: 0.649736\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614505\tvalid_0's auc: 0.770158\n",
      "[3]\tvalid_0's binary_logloss: 0.585713\tvalid_0's auc: 0.770649\n",
      "[4]\tvalid_0's binary_logloss: 0.561683\tvalid_0's auc: 0.772634\n",
      "[5]\tvalid_0's binary_logloss: 0.541646\tvalid_0's auc: 0.775379\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[7]\tvalid_0's binary_logloss: 0.511156\tvalid_0's auc: 0.77508\n",
      "[8]\tvalid_0's binary_logloss: 0.499351\tvalid_0's auc: 0.774687\n",
      "[9]\tvalid_0's binary_logloss: 0.489667\tvalid_0's auc: 0.773456\n",
      "[10]\tvalid_0's binary_logloss: 0.481189\tvalid_0's auc: 0.773241\n",
      "[11]\tvalid_0's binary_logloss: 0.473991\tvalid_0's auc: 0.773325\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[1]\tvalid_0's binary_logloss: 0.649987\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614878\tvalid_0's auc: 0.765334\n",
      "[3]\tvalid_0's binary_logloss: 0.585872\tvalid_0's auc: 0.769379\n",
      "[4]\tvalid_0's binary_logloss: 0.561615\tvalid_0's auc: 0.770104\n",
      "[5]\tvalid_0's binary_logloss: 0.541663\tvalid_0's auc: 0.77207\n",
      "[6]\tvalid_0's binary_logloss: 0.525011\tvalid_0's auc: 0.771646\n",
      "[7]\tvalid_0's binary_logloss: 0.511302\tvalid_0's auc: 0.770342\n",
      "[8]\tvalid_0's binary_logloss: 0.499137\tvalid_0's auc: 0.772387\n",
      "[9]\tvalid_0's binary_logloss: 0.488921\tvalid_0's auc: 0.772538\n",
      "[10]\tvalid_0's binary_logloss: 0.480526\tvalid_0's auc: 0.77215\n",
      "[11]\tvalid_0's binary_logloss: 0.473288\tvalid_0's auc: 0.772052\n",
      "[12]\tvalid_0's binary_logloss: 0.467178\tvalid_0's auc: 0.772153\n",
      "[13]\tvalid_0's binary_logloss: 0.461907\tvalid_0's auc: 0.772628\n",
      "[14]\tvalid_0's binary_logloss: 0.457719\tvalid_0's auc: 0.772073\n",
      "[15]\tvalid_0's binary_logloss: 0.454001\tvalid_0's auc: 0.772137\n",
      "[16]\tvalid_0's binary_logloss: 0.450598\tvalid_0's auc: 0.772892\n",
      "[17]\tvalid_0's binary_logloss: 0.447852\tvalid_0's auc: 0.773144\n",
      "[18]\tvalid_0's binary_logloss: 0.44543\tvalid_0's auc: 0.773478\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "[21]\tvalid_0's binary_logloss: 0.440773\tvalid_0's auc: 0.772489\n",
      "[22]\tvalid_0's binary_logloss: 0.439848\tvalid_0's auc: 0.771572\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[22]\tvalid_0's binary_logloss: 0.439848\tvalid_0's auc: 0.771572\n",
      "[1]\tvalid_0's binary_logloss: 0.650046\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614805\tvalid_0's auc: 0.768218\n",
      "[3]\tvalid_0's binary_logloss: 0.585989\tvalid_0's auc: 0.766994\n",
      "[4]\tvalid_0's binary_logloss: 0.56196\tvalid_0's auc: 0.767616\n",
      "[5]\tvalid_0's binary_logloss: 0.541971\tvalid_0's auc: 0.768534\n",
      "[6]\tvalid_0's binary_logloss: 0.525079\tvalid_0's auc: 0.769288\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[8]\tvalid_0's binary_logloss: 0.499175\tvalid_0's auc: 0.770306\n",
      "[9]\tvalid_0's binary_logloss: 0.489026\tvalid_0's auc: 0.770181\n",
      "[10]\tvalid_0's binary_logloss: 0.480527\tvalid_0's auc: 0.770099\n",
      "[11]\tvalid_0's binary_logloss: 0.47319\tvalid_0's auc: 0.770482\n",
      "[12]\tvalid_0's binary_logloss: 0.46736\tvalid_0's auc: 0.770093\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[1]\tvalid_0's binary_logloss: 0.649736\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614505\tvalid_0's auc: 0.770158\n",
      "[3]\tvalid_0's binary_logloss: 0.585713\tvalid_0's auc: 0.770649\n",
      "[4]\tvalid_0's binary_logloss: 0.561683\tvalid_0's auc: 0.772634\n",
      "[5]\tvalid_0's binary_logloss: 0.541646\tvalid_0's auc: 0.775379\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[7]\tvalid_0's binary_logloss: 0.511156\tvalid_0's auc: 0.77508\n",
      "[8]\tvalid_0's binary_logloss: 0.499351\tvalid_0's auc: 0.774687\n",
      "[9]\tvalid_0's binary_logloss: 0.489667\tvalid_0's auc: 0.773456\n",
      "[10]\tvalid_0's binary_logloss: 0.481189\tvalid_0's auc: 0.773241\n",
      "[11]\tvalid_0's binary_logloss: 0.473991\tvalid_0's auc: 0.773325\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[1]\tvalid_0's binary_logloss: 0.649987\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614878\tvalid_0's auc: 0.765334\n",
      "[3]\tvalid_0's binary_logloss: 0.585872\tvalid_0's auc: 0.769379\n",
      "[4]\tvalid_0's binary_logloss: 0.561615\tvalid_0's auc: 0.770104\n",
      "[5]\tvalid_0's binary_logloss: 0.541663\tvalid_0's auc: 0.77207\n",
      "[6]\tvalid_0's binary_logloss: 0.525011\tvalid_0's auc: 0.771646\n",
      "[7]\tvalid_0's binary_logloss: 0.511302\tvalid_0's auc: 0.770342\n",
      "[8]\tvalid_0's binary_logloss: 0.499137\tvalid_0's auc: 0.772387\n",
      "[9]\tvalid_0's binary_logloss: 0.488921\tvalid_0's auc: 0.772538\n",
      "[10]\tvalid_0's binary_logloss: 0.480526\tvalid_0's auc: 0.77215\n",
      "[11]\tvalid_0's binary_logloss: 0.473288\tvalid_0's auc: 0.772052\n",
      "[12]\tvalid_0's binary_logloss: 0.467178\tvalid_0's auc: 0.772153\n",
      "[13]\tvalid_0's binary_logloss: 0.461907\tvalid_0's auc: 0.772628\n",
      "[14]\tvalid_0's binary_logloss: 0.457719\tvalid_0's auc: 0.772073\n",
      "[15]\tvalid_0's binary_logloss: 0.454001\tvalid_0's auc: 0.772137\n",
      "[16]\tvalid_0's binary_logloss: 0.450598\tvalid_0's auc: 0.772892\n",
      "[17]\tvalid_0's binary_logloss: 0.447852\tvalid_0's auc: 0.773144\n",
      "[18]\tvalid_0's binary_logloss: 0.44543\tvalid_0's auc: 0.773478\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "[21]\tvalid_0's binary_logloss: 0.440773\tvalid_0's auc: 0.772489\n",
      "[22]\tvalid_0's binary_logloss: 0.439848\tvalid_0's auc: 0.771572\n",
      "[23]\tvalid_0's binary_logloss: 0.438724\tvalid_0's auc: 0.772306\n",
      "[24]\tvalid_0's binary_logloss: 0.438093\tvalid_0's auc: 0.772286\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[24]\tvalid_0's binary_logloss: 0.438093\tvalid_0's auc: 0.772286\n",
      "[1]\tvalid_0's binary_logloss: 0.650046\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614805\tvalid_0's auc: 0.768218\n",
      "[3]\tvalid_0's binary_logloss: 0.585989\tvalid_0's auc: 0.766994\n",
      "[4]\tvalid_0's binary_logloss: 0.56196\tvalid_0's auc: 0.767616\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5]\tvalid_0's binary_logloss: 0.541971\tvalid_0's auc: 0.768534\n",
      "[6]\tvalid_0's binary_logloss: 0.525079\tvalid_0's auc: 0.769288\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[8]\tvalid_0's binary_logloss: 0.499175\tvalid_0's auc: 0.770306\n",
      "[9]\tvalid_0's binary_logloss: 0.489026\tvalid_0's auc: 0.770181\n",
      "[10]\tvalid_0's binary_logloss: 0.480527\tvalid_0's auc: 0.770099\n",
      "[11]\tvalid_0's binary_logloss: 0.47319\tvalid_0's auc: 0.770482\n",
      "[12]\tvalid_0's binary_logloss: 0.46736\tvalid_0's auc: 0.770093\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[1]\tvalid_0's binary_logloss: 0.649736\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614505\tvalid_0's auc: 0.770158\n",
      "[3]\tvalid_0's binary_logloss: 0.585713\tvalid_0's auc: 0.770649\n",
      "[4]\tvalid_0's binary_logloss: 0.561683\tvalid_0's auc: 0.772634\n",
      "[5]\tvalid_0's binary_logloss: 0.541646\tvalid_0's auc: 0.775379\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[7]\tvalid_0's binary_logloss: 0.511156\tvalid_0's auc: 0.77508\n",
      "[8]\tvalid_0's binary_logloss: 0.499351\tvalid_0's auc: 0.774687\n",
      "[9]\tvalid_0's binary_logloss: 0.489667\tvalid_0's auc: 0.773456\n",
      "[10]\tvalid_0's binary_logloss: 0.481189\tvalid_0's auc: 0.773241\n",
      "[11]\tvalid_0's binary_logloss: 0.473991\tvalid_0's auc: 0.773325\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[1]\tvalid_0's binary_logloss: 0.649987\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614878\tvalid_0's auc: 0.765334\n",
      "[3]\tvalid_0's binary_logloss: 0.585872\tvalid_0's auc: 0.769379\n",
      "[4]\tvalid_0's binary_logloss: 0.561615\tvalid_0's auc: 0.770104\n",
      "[5]\tvalid_0's binary_logloss: 0.541663\tvalid_0's auc: 0.77207\n",
      "[6]\tvalid_0's binary_logloss: 0.525011\tvalid_0's auc: 0.771646\n",
      "[7]\tvalid_0's binary_logloss: 0.511302\tvalid_0's auc: 0.770342\n",
      "[8]\tvalid_0's binary_logloss: 0.499137\tvalid_0's auc: 0.772387\n",
      "[9]\tvalid_0's binary_logloss: 0.488921\tvalid_0's auc: 0.772538\n",
      "[10]\tvalid_0's binary_logloss: 0.480526\tvalid_0's auc: 0.77215\n",
      "[11]\tvalid_0's binary_logloss: 0.473288\tvalid_0's auc: 0.772052\n",
      "[12]\tvalid_0's binary_logloss: 0.467178\tvalid_0's auc: 0.772153\n",
      "[13]\tvalid_0's binary_logloss: 0.461907\tvalid_0's auc: 0.772628\n",
      "[14]\tvalid_0's binary_logloss: 0.457719\tvalid_0's auc: 0.772073\n",
      "[15]\tvalid_0's binary_logloss: 0.454001\tvalid_0's auc: 0.772137\n",
      "[16]\tvalid_0's binary_logloss: 0.450598\tvalid_0's auc: 0.772892\n",
      "[17]\tvalid_0's binary_logloss: 0.447852\tvalid_0's auc: 0.773144\n",
      "[18]\tvalid_0's binary_logloss: 0.44543\tvalid_0's auc: 0.773478\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "[21]\tvalid_0's binary_logloss: 0.440773\tvalid_0's auc: 0.772489\n",
      "[22]\tvalid_0's binary_logloss: 0.439848\tvalid_0's auc: 0.771572\n",
      "[23]\tvalid_0's binary_logloss: 0.438724\tvalid_0's auc: 0.772306\n",
      "[24]\tvalid_0's binary_logloss: 0.438093\tvalid_0's auc: 0.772286\n",
      "Early stopping, best iteration is:\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[1]\tvalid_0's binary_logloss: 0.650046\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614805\tvalid_0's auc: 0.768218\n",
      "[3]\tvalid_0's binary_logloss: 0.585989\tvalid_0's auc: 0.766994\n",
      "[4]\tvalid_0's binary_logloss: 0.56196\tvalid_0's auc: 0.767616\n",
      "[5]\tvalid_0's binary_logloss: 0.541971\tvalid_0's auc: 0.768534\n",
      "[6]\tvalid_0's binary_logloss: 0.525079\tvalid_0's auc: 0.769288\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[8]\tvalid_0's binary_logloss: 0.499175\tvalid_0's auc: 0.770306\n",
      "[9]\tvalid_0's binary_logloss: 0.489026\tvalid_0's auc: 0.770181\n",
      "[10]\tvalid_0's binary_logloss: 0.480527\tvalid_0's auc: 0.770099\n",
      "[11]\tvalid_0's binary_logloss: 0.47319\tvalid_0's auc: 0.770482\n",
      "[12]\tvalid_0's binary_logloss: 0.46736\tvalid_0's auc: 0.770093\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[1]\tvalid_0's binary_logloss: 0.649736\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614505\tvalid_0's auc: 0.770158\n",
      "[3]\tvalid_0's binary_logloss: 0.585713\tvalid_0's auc: 0.770649\n",
      "[4]\tvalid_0's binary_logloss: 0.561683\tvalid_0's auc: 0.772634\n",
      "[5]\tvalid_0's binary_logloss: 0.541646\tvalid_0's auc: 0.775379\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[7]\tvalid_0's binary_logloss: 0.511156\tvalid_0's auc: 0.77508\n",
      "[8]\tvalid_0's binary_logloss: 0.499351\tvalid_0's auc: 0.774687\n",
      "[9]\tvalid_0's binary_logloss: 0.489667\tvalid_0's auc: 0.773456\n",
      "[10]\tvalid_0's binary_logloss: 0.481189\tvalid_0's auc: 0.773241\n",
      "[11]\tvalid_0's binary_logloss: 0.473991\tvalid_0's auc: 0.773325\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[1]\tvalid_0's binary_logloss: 0.649987\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614878\tvalid_0's auc: 0.765334\n",
      "[3]\tvalid_0's binary_logloss: 0.585872\tvalid_0's auc: 0.769379\n",
      "[4]\tvalid_0's binary_logloss: 0.561615\tvalid_0's auc: 0.770104\n",
      "[5]\tvalid_0's binary_logloss: 0.541663\tvalid_0's auc: 0.77207\n",
      "[6]\tvalid_0's binary_logloss: 0.525011\tvalid_0's auc: 0.771646\n",
      "[7]\tvalid_0's binary_logloss: 0.511302\tvalid_0's auc: 0.770342\n",
      "[8]\tvalid_0's binary_logloss: 0.499137\tvalid_0's auc: 0.772387\n",
      "[9]\tvalid_0's binary_logloss: 0.488921\tvalid_0's auc: 0.772538\n",
      "[10]\tvalid_0's binary_logloss: 0.480526\tvalid_0's auc: 0.77215\n",
      "[11]\tvalid_0's binary_logloss: 0.473288\tvalid_0's auc: 0.772052\n",
      "[12]\tvalid_0's binary_logloss: 0.467178\tvalid_0's auc: 0.772153\n",
      "[13]\tvalid_0's binary_logloss: 0.461907\tvalid_0's auc: 0.772628\n",
      "[14]\tvalid_0's binary_logloss: 0.457719\tvalid_0's auc: 0.772073\n",
      "[15]\tvalid_0's binary_logloss: 0.454001\tvalid_0's auc: 0.772137\n",
      "[16]\tvalid_0's binary_logloss: 0.450598\tvalid_0's auc: 0.772892\n",
      "[17]\tvalid_0's binary_logloss: 0.447852\tvalid_0's auc: 0.773144\n",
      "[18]\tvalid_0's binary_logloss: 0.44543\tvalid_0's auc: 0.773478\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "[21]\tvalid_0's binary_logloss: 0.440773\tvalid_0's auc: 0.772489\n",
      "[22]\tvalid_0's binary_logloss: 0.439848\tvalid_0's auc: 0.771572\n",
      "[23]\tvalid_0's binary_logloss: 0.438724\tvalid_0's auc: 0.772306\n",
      "[24]\tvalid_0's binary_logloss: 0.438093\tvalid_0's auc: 0.772286\n",
      "Early stopping, best iteration is:\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[1]\tvalid_0's binary_logloss: 0.650046\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614805\tvalid_0's auc: 0.768218\n",
      "[3]\tvalid_0's binary_logloss: 0.585989\tvalid_0's auc: 0.766994\n",
      "[4]\tvalid_0's binary_logloss: 0.56196\tvalid_0's auc: 0.767616\n",
      "[5]\tvalid_0's binary_logloss: 0.541971\tvalid_0's auc: 0.768534\n",
      "[6]\tvalid_0's binary_logloss: 0.525079\tvalid_0's auc: 0.769288\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[8]\tvalid_0's binary_logloss: 0.499175\tvalid_0's auc: 0.770306\n",
      "[9]\tvalid_0's binary_logloss: 0.489026\tvalid_0's auc: 0.770181\n",
      "[10]\tvalid_0's binary_logloss: 0.480527\tvalid_0's auc: 0.770099\n",
      "[11]\tvalid_0's binary_logloss: 0.47319\tvalid_0's auc: 0.770482\n",
      "[12]\tvalid_0's binary_logloss: 0.46736\tvalid_0's auc: 0.770093\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[1]\tvalid_0's binary_logloss: 0.649736\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614505\tvalid_0's auc: 0.770158\n",
      "[3]\tvalid_0's binary_logloss: 0.585713\tvalid_0's auc: 0.770649\n",
      "[4]\tvalid_0's binary_logloss: 0.561683\tvalid_0's auc: 0.772634\n",
      "[5]\tvalid_0's binary_logloss: 0.541646\tvalid_0's auc: 0.775379\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[7]\tvalid_0's binary_logloss: 0.511156\tvalid_0's auc: 0.77508\n",
      "[8]\tvalid_0's binary_logloss: 0.499351\tvalid_0's auc: 0.774687\n",
      "[9]\tvalid_0's binary_logloss: 0.489667\tvalid_0's auc: 0.773456\n",
      "[10]\tvalid_0's binary_logloss: 0.481189\tvalid_0's auc: 0.773241\n",
      "[11]\tvalid_0's binary_logloss: 0.473991\tvalid_0's auc: 0.773325\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[1]\tvalid_0's binary_logloss: 0.649987\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614878\tvalid_0's auc: 0.765334\n",
      "[3]\tvalid_0's binary_logloss: 0.585872\tvalid_0's auc: 0.769379\n",
      "[4]\tvalid_0's binary_logloss: 0.561615\tvalid_0's auc: 0.770104\n",
      "[5]\tvalid_0's binary_logloss: 0.541663\tvalid_0's auc: 0.77207\n",
      "[6]\tvalid_0's binary_logloss: 0.525011\tvalid_0's auc: 0.771646\n",
      "[7]\tvalid_0's binary_logloss: 0.511302\tvalid_0's auc: 0.770342\n",
      "[8]\tvalid_0's binary_logloss: 0.499137\tvalid_0's auc: 0.772387\n",
      "[9]\tvalid_0's binary_logloss: 0.488921\tvalid_0's auc: 0.772538\n",
      "[10]\tvalid_0's binary_logloss: 0.480526\tvalid_0's auc: 0.77215\n",
      "[11]\tvalid_0's binary_logloss: 0.473288\tvalid_0's auc: 0.772052\n",
      "[12]\tvalid_0's binary_logloss: 0.467178\tvalid_0's auc: 0.772153\n",
      "[13]\tvalid_0's binary_logloss: 0.461907\tvalid_0's auc: 0.772628\n",
      "[14]\tvalid_0's binary_logloss: 0.457719\tvalid_0's auc: 0.772073\n",
      "[15]\tvalid_0's binary_logloss: 0.454001\tvalid_0's auc: 0.772137\n",
      "[16]\tvalid_0's binary_logloss: 0.450598\tvalid_0's auc: 0.772892\n",
      "[17]\tvalid_0's binary_logloss: 0.447852\tvalid_0's auc: 0.773144\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18]\tvalid_0's binary_logloss: 0.44543\tvalid_0's auc: 0.773478\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "[21]\tvalid_0's binary_logloss: 0.440773\tvalid_0's auc: 0.772489\n",
      "[22]\tvalid_0's binary_logloss: 0.439848\tvalid_0's auc: 0.771572\n",
      "[23]\tvalid_0's binary_logloss: 0.438724\tvalid_0's auc: 0.772306\n",
      "[24]\tvalid_0's binary_logloss: 0.438093\tvalid_0's auc: 0.772286\n",
      "Early stopping, best iteration is:\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[1]\tvalid_0's binary_logloss: 0.650046\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614805\tvalid_0's auc: 0.768218\n",
      "[3]\tvalid_0's binary_logloss: 0.585989\tvalid_0's auc: 0.766994\n",
      "[4]\tvalid_0's binary_logloss: 0.56196\tvalid_0's auc: 0.767616\n",
      "[5]\tvalid_0's binary_logloss: 0.541971\tvalid_0's auc: 0.768534\n",
      "[6]\tvalid_0's binary_logloss: 0.525079\tvalid_0's auc: 0.769288\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[8]\tvalid_0's binary_logloss: 0.499175\tvalid_0's auc: 0.770306\n",
      "[9]\tvalid_0's binary_logloss: 0.489026\tvalid_0's auc: 0.770181\n",
      "[10]\tvalid_0's binary_logloss: 0.480527\tvalid_0's auc: 0.770099\n",
      "[11]\tvalid_0's binary_logloss: 0.47319\tvalid_0's auc: 0.770482\n",
      "[12]\tvalid_0's binary_logloss: 0.46736\tvalid_0's auc: 0.770093\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[1]\tvalid_0's binary_logloss: 0.649736\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614505\tvalid_0's auc: 0.770158\n",
      "[3]\tvalid_0's binary_logloss: 0.585713\tvalid_0's auc: 0.770649\n",
      "[4]\tvalid_0's binary_logloss: 0.561683\tvalid_0's auc: 0.772634\n",
      "[5]\tvalid_0's binary_logloss: 0.541646\tvalid_0's auc: 0.775379\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[7]\tvalid_0's binary_logloss: 0.511156\tvalid_0's auc: 0.77508\n",
      "[8]\tvalid_0's binary_logloss: 0.499351\tvalid_0's auc: 0.774687\n",
      "[9]\tvalid_0's binary_logloss: 0.489667\tvalid_0's auc: 0.773456\n",
      "[10]\tvalid_0's binary_logloss: 0.481189\tvalid_0's auc: 0.773241\n",
      "[11]\tvalid_0's binary_logloss: 0.473991\tvalid_0's auc: 0.773325\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[1]\tvalid_0's binary_logloss: 0.649987\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614878\tvalid_0's auc: 0.765334\n",
      "[3]\tvalid_0's binary_logloss: 0.585872\tvalid_0's auc: 0.769379\n",
      "[4]\tvalid_0's binary_logloss: 0.561615\tvalid_0's auc: 0.770104\n",
      "[5]\tvalid_0's binary_logloss: 0.541663\tvalid_0's auc: 0.77207\n",
      "[6]\tvalid_0's binary_logloss: 0.525011\tvalid_0's auc: 0.771646\n",
      "[7]\tvalid_0's binary_logloss: 0.511302\tvalid_0's auc: 0.770342\n",
      "[8]\tvalid_0's binary_logloss: 0.499137\tvalid_0's auc: 0.772387\n",
      "[9]\tvalid_0's binary_logloss: 0.488921\tvalid_0's auc: 0.772538\n",
      "[10]\tvalid_0's binary_logloss: 0.480526\tvalid_0's auc: 0.77215\n",
      "[11]\tvalid_0's binary_logloss: 0.473288\tvalid_0's auc: 0.772052\n",
      "[12]\tvalid_0's binary_logloss: 0.467178\tvalid_0's auc: 0.772153\n",
      "[13]\tvalid_0's binary_logloss: 0.461907\tvalid_0's auc: 0.772628\n",
      "[14]\tvalid_0's binary_logloss: 0.457719\tvalid_0's auc: 0.772073\n",
      "[15]\tvalid_0's binary_logloss: 0.454001\tvalid_0's auc: 0.772137\n",
      "[16]\tvalid_0's binary_logloss: 0.450598\tvalid_0's auc: 0.772892\n",
      "[17]\tvalid_0's binary_logloss: 0.447852\tvalid_0's auc: 0.773144\n",
      "[18]\tvalid_0's binary_logloss: 0.44543\tvalid_0's auc: 0.773478\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "[21]\tvalid_0's binary_logloss: 0.440773\tvalid_0's auc: 0.772489\n",
      "[22]\tvalid_0's binary_logloss: 0.439848\tvalid_0's auc: 0.771572\n",
      "[23]\tvalid_0's binary_logloss: 0.438724\tvalid_0's auc: 0.772306\n",
      "[24]\tvalid_0's binary_logloss: 0.438093\tvalid_0's auc: 0.772286\n",
      "Early stopping, best iteration is:\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[1]\tvalid_0's binary_logloss: 0.650046\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614805\tvalid_0's auc: 0.768218\n",
      "[3]\tvalid_0's binary_logloss: 0.585989\tvalid_0's auc: 0.766994\n",
      "[4]\tvalid_0's binary_logloss: 0.56196\tvalid_0's auc: 0.767616\n",
      "[5]\tvalid_0's binary_logloss: 0.541971\tvalid_0's auc: 0.768534\n",
      "[6]\tvalid_0's binary_logloss: 0.525079\tvalid_0's auc: 0.769288\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[8]\tvalid_0's binary_logloss: 0.499175\tvalid_0's auc: 0.770306\n",
      "[9]\tvalid_0's binary_logloss: 0.489026\tvalid_0's auc: 0.770181\n",
      "[10]\tvalid_0's binary_logloss: 0.480527\tvalid_0's auc: 0.770099\n",
      "[11]\tvalid_0's binary_logloss: 0.47319\tvalid_0's auc: 0.770482\n",
      "[12]\tvalid_0's binary_logloss: 0.46736\tvalid_0's auc: 0.770093\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[1]\tvalid_0's binary_logloss: 0.649736\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614505\tvalid_0's auc: 0.770158\n",
      "[3]\tvalid_0's binary_logloss: 0.585713\tvalid_0's auc: 0.770649\n",
      "[4]\tvalid_0's binary_logloss: 0.561683\tvalid_0's auc: 0.772634\n",
      "[5]\tvalid_0's binary_logloss: 0.541646\tvalid_0's auc: 0.775379\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[7]\tvalid_0's binary_logloss: 0.511156\tvalid_0's auc: 0.77508\n",
      "[8]\tvalid_0's binary_logloss: 0.499351\tvalid_0's auc: 0.774687\n",
      "[9]\tvalid_0's binary_logloss: 0.489667\tvalid_0's auc: 0.773456\n",
      "[10]\tvalid_0's binary_logloss: 0.481189\tvalid_0's auc: 0.773241\n",
      "[11]\tvalid_0's binary_logloss: 0.473991\tvalid_0's auc: 0.773325\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[1]\tvalid_0's binary_logloss: 0.649987\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614878\tvalid_0's auc: 0.765334\n",
      "[3]\tvalid_0's binary_logloss: 0.585872\tvalid_0's auc: 0.769379\n",
      "[4]\tvalid_0's binary_logloss: 0.561615\tvalid_0's auc: 0.770104\n",
      "[5]\tvalid_0's binary_logloss: 0.541663\tvalid_0's auc: 0.77207\n",
      "[6]\tvalid_0's binary_logloss: 0.525011\tvalid_0's auc: 0.771646\n",
      "[7]\tvalid_0's binary_logloss: 0.511302\tvalid_0's auc: 0.770342\n",
      "[8]\tvalid_0's binary_logloss: 0.499137\tvalid_0's auc: 0.772387\n",
      "[9]\tvalid_0's binary_logloss: 0.488921\tvalid_0's auc: 0.772538\n",
      "[10]\tvalid_0's binary_logloss: 0.480526\tvalid_0's auc: 0.77215\n",
      "[11]\tvalid_0's binary_logloss: 0.473288\tvalid_0's auc: 0.772052\n",
      "[12]\tvalid_0's binary_logloss: 0.467178\tvalid_0's auc: 0.772153\n",
      "[13]\tvalid_0's binary_logloss: 0.461907\tvalid_0's auc: 0.772628\n",
      "[14]\tvalid_0's binary_logloss: 0.457719\tvalid_0's auc: 0.772073\n",
      "[15]\tvalid_0's binary_logloss: 0.454001\tvalid_0's auc: 0.772137\n",
      "[16]\tvalid_0's binary_logloss: 0.450598\tvalid_0's auc: 0.772892\n",
      "[17]\tvalid_0's binary_logloss: 0.447852\tvalid_0's auc: 0.773144\n",
      "[18]\tvalid_0's binary_logloss: 0.44543\tvalid_0's auc: 0.773478\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[20]\tvalid_0's binary_logloss: 0.442102\tvalid_0's auc: 0.772553\n",
      "[21]\tvalid_0's binary_logloss: 0.440773\tvalid_0's auc: 0.772489\n",
      "[22]\tvalid_0's binary_logloss: 0.439848\tvalid_0's auc: 0.771572\n",
      "[23]\tvalid_0's binary_logloss: 0.438724\tvalid_0's auc: 0.772306\n",
      "[24]\tvalid_0's binary_logloss: 0.438093\tvalid_0's auc: 0.772286\n",
      "Early stopping, best iteration is:\n",
      "[19]\tvalid_0's binary_logloss: 0.443565\tvalid_0's auc: 0.773689\n",
      "[1]\tvalid_0's binary_logloss: 0.650046\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614805\tvalid_0's auc: 0.768218\n",
      "[3]\tvalid_0's binary_logloss: 0.585989\tvalid_0's auc: 0.766994\n",
      "[4]\tvalid_0's binary_logloss: 0.56196\tvalid_0's auc: 0.767616\n",
      "[5]\tvalid_0's binary_logloss: 0.541971\tvalid_0's auc: 0.768534\n",
      "[6]\tvalid_0's binary_logloss: 0.525079\tvalid_0's auc: 0.769288\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[8]\tvalid_0's binary_logloss: 0.499175\tvalid_0's auc: 0.770306\n",
      "[9]\tvalid_0's binary_logloss: 0.489026\tvalid_0's auc: 0.770181\n",
      "[10]\tvalid_0's binary_logloss: 0.480527\tvalid_0's auc: 0.770099\n",
      "[11]\tvalid_0's binary_logloss: 0.47319\tvalid_0's auc: 0.770482\n",
      "[12]\tvalid_0's binary_logloss: 0.46736\tvalid_0's auc: 0.770093\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.510989\tvalid_0's auc: 0.770722\n",
      "[1]\tvalid_0's binary_logloss: 0.649736\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614505\tvalid_0's auc: 0.770158\n",
      "[3]\tvalid_0's binary_logloss: 0.585713\tvalid_0's auc: 0.770649\n",
      "[4]\tvalid_0's binary_logloss: 0.561683\tvalid_0's auc: 0.772634\n",
      "[5]\tvalid_0's binary_logloss: 0.541646\tvalid_0's auc: 0.775379\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[7]\tvalid_0's binary_logloss: 0.511156\tvalid_0's auc: 0.77508\n",
      "[8]\tvalid_0's binary_logloss: 0.499351\tvalid_0's auc: 0.774687\n",
      "[9]\tvalid_0's binary_logloss: 0.489667\tvalid_0's auc: 0.773456\n",
      "[10]\tvalid_0's binary_logloss: 0.481189\tvalid_0's auc: 0.773241\n",
      "[11]\tvalid_0's binary_logloss: 0.473991\tvalid_0's auc: 0.773325\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.524719\tvalid_0's auc: 0.775956\n",
      "[1]\tvalid_0's binary_logloss: 0.639924\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598752\tvalid_0's auc: 0.766736\n",
      "[3]\tvalid_0's binary_logloss: 0.566652\tvalid_0's auc: 0.768039\n",
      "[4]\tvalid_0's binary_logloss: 0.540753\tvalid_0's auc: 0.771686\n",
      "[5]\tvalid_0's binary_logloss: 0.520077\tvalid_0's auc: 0.772744\n",
      "[6]\tvalid_0's binary_logloss: 0.503503\tvalid_0's auc: 0.774082\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[8]\tvalid_0's binary_logloss: 0.4795\tvalid_0's auc: 0.77261\n",
      "[9]\tvalid_0's binary_logloss: 0.470271\tvalid_0's auc: 0.773885\n",
      "[10]\tvalid_0's binary_logloss: 0.463132\tvalid_0's auc: 0.773888\n",
      "[11]\tvalid_0's binary_logloss: 0.457223\tvalid_0's auc: 0.772738\n",
      "[12]\tvalid_0's binary_logloss: 0.452691\tvalid_0's auc: 0.772988\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.639997\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598647\tvalid_0's auc: 0.768051\n",
      "[3]\tvalid_0's binary_logloss: 0.56668\tvalid_0's auc: 0.769974\n",
      "[4]\tvalid_0's binary_logloss: 0.541144\tvalid_0's auc: 0.770928\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[6]\tvalid_0's binary_logloss: 0.504237\tvalid_0's auc: 0.771318\n",
      "[7]\tvalid_0's binary_logloss: 0.490907\tvalid_0's auc: 0.770627\n",
      "[8]\tvalid_0's binary_logloss: 0.48037\tvalid_0's auc: 0.769352\n",
      "[9]\tvalid_0's binary_logloss: 0.471982\tvalid_0's auc: 0.768651\n",
      "[10]\tvalid_0's binary_logloss: 0.464704\tvalid_0's auc: 0.769834\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[1]\tvalid_0's binary_logloss: 0.639605\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598235\tvalid_0's auc: 0.770893\n",
      "[3]\tvalid_0's binary_logloss: 0.566282\tvalid_0's auc: 0.772481\n",
      "[4]\tvalid_0's binary_logloss: 0.540521\tvalid_0's auc: 0.774474\n",
      "[5]\tvalid_0's binary_logloss: 0.51982\tvalid_0's auc: 0.775178\n",
      "[6]\tvalid_0's binary_logloss: 0.503527\tvalid_0's auc: 0.774708\n",
      "[7]\tvalid_0's binary_logloss: 0.490433\tvalid_0's auc: 0.77386\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[9]\tvalid_0's binary_logloss: 0.47085\tvalid_0's auc: 0.774124\n",
      "[10]\tvalid_0's binary_logloss: 0.463677\tvalid_0's auc: 0.774711\n",
      "[11]\tvalid_0's binary_logloss: 0.458534\tvalid_0's auc: 0.773313\n",
      "[12]\tvalid_0's binary_logloss: 0.45358\tvalid_0's auc: 0.773877\n",
      "[13]\tvalid_0's binary_logloss: 0.449739\tvalid_0's auc: 0.773154\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[1]\tvalid_0's binary_logloss: 0.639924\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598752\tvalid_0's auc: 0.766736\n",
      "[3]\tvalid_0's binary_logloss: 0.566652\tvalid_0's auc: 0.768039\n",
      "[4]\tvalid_0's binary_logloss: 0.540753\tvalid_0's auc: 0.771686\n",
      "[5]\tvalid_0's binary_logloss: 0.520077\tvalid_0's auc: 0.772744\n",
      "[6]\tvalid_0's binary_logloss: 0.503503\tvalid_0's auc: 0.774082\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[8]\tvalid_0's binary_logloss: 0.4795\tvalid_0's auc: 0.77261\n",
      "[9]\tvalid_0's binary_logloss: 0.470271\tvalid_0's auc: 0.773885\n",
      "[10]\tvalid_0's binary_logloss: 0.463132\tvalid_0's auc: 0.773888\n",
      "[11]\tvalid_0's binary_logloss: 0.457223\tvalid_0's auc: 0.772738\n",
      "[12]\tvalid_0's binary_logloss: 0.452691\tvalid_0's auc: 0.772988\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[1]\tvalid_0's binary_logloss: 0.639997\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598647\tvalid_0's auc: 0.768051\n",
      "[3]\tvalid_0's binary_logloss: 0.56668\tvalid_0's auc: 0.769974\n",
      "[4]\tvalid_0's binary_logloss: 0.541144\tvalid_0's auc: 0.770928\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[6]\tvalid_0's binary_logloss: 0.504237\tvalid_0's auc: 0.771318\n",
      "[7]\tvalid_0's binary_logloss: 0.490907\tvalid_0's auc: 0.770627\n",
      "[8]\tvalid_0's binary_logloss: 0.48037\tvalid_0's auc: 0.769352\n",
      "[9]\tvalid_0's binary_logloss: 0.471982\tvalid_0's auc: 0.768651\n",
      "[10]\tvalid_0's binary_logloss: 0.464704\tvalid_0's auc: 0.769834\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[1]\tvalid_0's binary_logloss: 0.639605\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598235\tvalid_0's auc: 0.770893\n",
      "[3]\tvalid_0's binary_logloss: 0.566282\tvalid_0's auc: 0.772481\n",
      "[4]\tvalid_0's binary_logloss: 0.540521\tvalid_0's auc: 0.774474\n",
      "[5]\tvalid_0's binary_logloss: 0.51982\tvalid_0's auc: 0.775178\n",
      "[6]\tvalid_0's binary_logloss: 0.503527\tvalid_0's auc: 0.774708\n",
      "[7]\tvalid_0's binary_logloss: 0.490433\tvalid_0's auc: 0.77386\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[9]\tvalid_0's binary_logloss: 0.47085\tvalid_0's auc: 0.774124\n",
      "[10]\tvalid_0's binary_logloss: 0.463677\tvalid_0's auc: 0.774711\n",
      "[11]\tvalid_0's binary_logloss: 0.458534\tvalid_0's auc: 0.773313\n",
      "[12]\tvalid_0's binary_logloss: 0.45358\tvalid_0's auc: 0.773877\n",
      "[13]\tvalid_0's binary_logloss: 0.449739\tvalid_0's auc: 0.773154\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[1]\tvalid_0's binary_logloss: 0.639924\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598752\tvalid_0's auc: 0.766736\n",
      "[3]\tvalid_0's binary_logloss: 0.566652\tvalid_0's auc: 0.768039\n",
      "[4]\tvalid_0's binary_logloss: 0.540753\tvalid_0's auc: 0.771686\n",
      "[5]\tvalid_0's binary_logloss: 0.520077\tvalid_0's auc: 0.772744\n",
      "[6]\tvalid_0's binary_logloss: 0.503503\tvalid_0's auc: 0.774082\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[8]\tvalid_0's binary_logloss: 0.4795\tvalid_0's auc: 0.77261\n",
      "[9]\tvalid_0's binary_logloss: 0.470271\tvalid_0's auc: 0.773885\n",
      "[10]\tvalid_0's binary_logloss: 0.463132\tvalid_0's auc: 0.773888\n",
      "[11]\tvalid_0's binary_logloss: 0.457223\tvalid_0's auc: 0.772738\n",
      "[12]\tvalid_0's binary_logloss: 0.452691\tvalid_0's auc: 0.772988\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[1]\tvalid_0's binary_logloss: 0.639997\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598647\tvalid_0's auc: 0.768051\n",
      "[3]\tvalid_0's binary_logloss: 0.56668\tvalid_0's auc: 0.769974\n",
      "[4]\tvalid_0's binary_logloss: 0.541144\tvalid_0's auc: 0.770928\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[6]\tvalid_0's binary_logloss: 0.504237\tvalid_0's auc: 0.771318\n",
      "[7]\tvalid_0's binary_logloss: 0.490907\tvalid_0's auc: 0.770627\n",
      "[8]\tvalid_0's binary_logloss: 0.48037\tvalid_0's auc: 0.769352\n",
      "[9]\tvalid_0's binary_logloss: 0.471982\tvalid_0's auc: 0.768651\n",
      "[10]\tvalid_0's binary_logloss: 0.464704\tvalid_0's auc: 0.769834\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[1]\tvalid_0's binary_logloss: 0.639605\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598235\tvalid_0's auc: 0.770893\n",
      "[3]\tvalid_0's binary_logloss: 0.566282\tvalid_0's auc: 0.772481\n",
      "[4]\tvalid_0's binary_logloss: 0.540521\tvalid_0's auc: 0.774474\n",
      "[5]\tvalid_0's binary_logloss: 0.51982\tvalid_0's auc: 0.775178\n",
      "[6]\tvalid_0's binary_logloss: 0.503527\tvalid_0's auc: 0.774708\n",
      "[7]\tvalid_0's binary_logloss: 0.490433\tvalid_0's auc: 0.77386\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[9]\tvalid_0's binary_logloss: 0.47085\tvalid_0's auc: 0.774124\n",
      "[10]\tvalid_0's binary_logloss: 0.463677\tvalid_0's auc: 0.774711\n",
      "[11]\tvalid_0's binary_logloss: 0.458534\tvalid_0's auc: 0.773313\n",
      "[12]\tvalid_0's binary_logloss: 0.45358\tvalid_0's auc: 0.773877\n",
      "[13]\tvalid_0's binary_logloss: 0.449739\tvalid_0's auc: 0.773154\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[1]\tvalid_0's binary_logloss: 0.639924\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598752\tvalid_0's auc: 0.766736\n",
      "[3]\tvalid_0's binary_logloss: 0.566652\tvalid_0's auc: 0.768039\n",
      "[4]\tvalid_0's binary_logloss: 0.540753\tvalid_0's auc: 0.771686\n",
      "[5]\tvalid_0's binary_logloss: 0.520077\tvalid_0's auc: 0.772744\n",
      "[6]\tvalid_0's binary_logloss: 0.503503\tvalid_0's auc: 0.774082\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[8]\tvalid_0's binary_logloss: 0.4795\tvalid_0's auc: 0.77261\n",
      "[9]\tvalid_0's binary_logloss: 0.470271\tvalid_0's auc: 0.773885\n",
      "[10]\tvalid_0's binary_logloss: 0.463132\tvalid_0's auc: 0.773888\n",
      "[11]\tvalid_0's binary_logloss: 0.457223\tvalid_0's auc: 0.772738\n",
      "[12]\tvalid_0's binary_logloss: 0.452691\tvalid_0's auc: 0.772988\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[1]\tvalid_0's binary_logloss: 0.639997\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598647\tvalid_0's auc: 0.768051\n",
      "[3]\tvalid_0's binary_logloss: 0.56668\tvalid_0's auc: 0.769974\n",
      "[4]\tvalid_0's binary_logloss: 0.541144\tvalid_0's auc: 0.770928\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[6]\tvalid_0's binary_logloss: 0.504237\tvalid_0's auc: 0.771318\n",
      "[7]\tvalid_0's binary_logloss: 0.490907\tvalid_0's auc: 0.770627\n",
      "[8]\tvalid_0's binary_logloss: 0.48037\tvalid_0's auc: 0.769352\n",
      "[9]\tvalid_0's binary_logloss: 0.471982\tvalid_0's auc: 0.768651\n",
      "[10]\tvalid_0's binary_logloss: 0.464704\tvalid_0's auc: 0.769834\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.639605\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598235\tvalid_0's auc: 0.770893\n",
      "[3]\tvalid_0's binary_logloss: 0.566282\tvalid_0's auc: 0.772481\n",
      "[4]\tvalid_0's binary_logloss: 0.540521\tvalid_0's auc: 0.774474\n",
      "[5]\tvalid_0's binary_logloss: 0.51982\tvalid_0's auc: 0.775178\n",
      "[6]\tvalid_0's binary_logloss: 0.503527\tvalid_0's auc: 0.774708\n",
      "[7]\tvalid_0's binary_logloss: 0.490433\tvalid_0's auc: 0.77386\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[9]\tvalid_0's binary_logloss: 0.47085\tvalid_0's auc: 0.774124\n",
      "[10]\tvalid_0's binary_logloss: 0.463677\tvalid_0's auc: 0.774711\n",
      "[11]\tvalid_0's binary_logloss: 0.458534\tvalid_0's auc: 0.773313\n",
      "[12]\tvalid_0's binary_logloss: 0.45358\tvalid_0's auc: 0.773877\n",
      "[13]\tvalid_0's binary_logloss: 0.449739\tvalid_0's auc: 0.773154\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[1]\tvalid_0's binary_logloss: 0.639924\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598752\tvalid_0's auc: 0.766736\n",
      "[3]\tvalid_0's binary_logloss: 0.566652\tvalid_0's auc: 0.768039\n",
      "[4]\tvalid_0's binary_logloss: 0.540753\tvalid_0's auc: 0.771686\n",
      "[5]\tvalid_0's binary_logloss: 0.520077\tvalid_0's auc: 0.772744\n",
      "[6]\tvalid_0's binary_logloss: 0.503503\tvalid_0's auc: 0.774082\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[8]\tvalid_0's binary_logloss: 0.4795\tvalid_0's auc: 0.77261\n",
      "[9]\tvalid_0's binary_logloss: 0.470271\tvalid_0's auc: 0.773885\n",
      "[10]\tvalid_0's binary_logloss: 0.463132\tvalid_0's auc: 0.773888\n",
      "[11]\tvalid_0's binary_logloss: 0.457223\tvalid_0's auc: 0.772738\n",
      "[12]\tvalid_0's binary_logloss: 0.452691\tvalid_0's auc: 0.772988\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[1]\tvalid_0's binary_logloss: 0.639997\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598647\tvalid_0's auc: 0.768051\n",
      "[3]\tvalid_0's binary_logloss: 0.56668\tvalid_0's auc: 0.769974\n",
      "[4]\tvalid_0's binary_logloss: 0.541144\tvalid_0's auc: 0.770928\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[6]\tvalid_0's binary_logloss: 0.504237\tvalid_0's auc: 0.771318\n",
      "[7]\tvalid_0's binary_logloss: 0.490907\tvalid_0's auc: 0.770627\n",
      "[8]\tvalid_0's binary_logloss: 0.48037\tvalid_0's auc: 0.769352\n",
      "[9]\tvalid_0's binary_logloss: 0.471982\tvalid_0's auc: 0.768651\n",
      "[10]\tvalid_0's binary_logloss: 0.464704\tvalid_0's auc: 0.769834\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[1]\tvalid_0's binary_logloss: 0.639605\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598235\tvalid_0's auc: 0.770893\n",
      "[3]\tvalid_0's binary_logloss: 0.566282\tvalid_0's auc: 0.772481\n",
      "[4]\tvalid_0's binary_logloss: 0.540521\tvalid_0's auc: 0.774474\n",
      "[5]\tvalid_0's binary_logloss: 0.51982\tvalid_0's auc: 0.775178\n",
      "[6]\tvalid_0's binary_logloss: 0.503527\tvalid_0's auc: 0.774708\n",
      "[7]\tvalid_0's binary_logloss: 0.490433\tvalid_0's auc: 0.77386\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[9]\tvalid_0's binary_logloss: 0.47085\tvalid_0's auc: 0.774124\n",
      "[10]\tvalid_0's binary_logloss: 0.463677\tvalid_0's auc: 0.774711\n",
      "[11]\tvalid_0's binary_logloss: 0.458534\tvalid_0's auc: 0.773313\n",
      "[12]\tvalid_0's binary_logloss: 0.45358\tvalid_0's auc: 0.773877\n",
      "[13]\tvalid_0's binary_logloss: 0.449739\tvalid_0's auc: 0.773154\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[1]\tvalid_0's binary_logloss: 0.639924\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598752\tvalid_0's auc: 0.766736\n",
      "[3]\tvalid_0's binary_logloss: 0.566652\tvalid_0's auc: 0.768039\n",
      "[4]\tvalid_0's binary_logloss: 0.540753\tvalid_0's auc: 0.771686\n",
      "[5]\tvalid_0's binary_logloss: 0.520077\tvalid_0's auc: 0.772744\n",
      "[6]\tvalid_0's binary_logloss: 0.503503\tvalid_0's auc: 0.774082\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[8]\tvalid_0's binary_logloss: 0.4795\tvalid_0's auc: 0.77261\n",
      "[9]\tvalid_0's binary_logloss: 0.470271\tvalid_0's auc: 0.773885\n",
      "[10]\tvalid_0's binary_logloss: 0.463132\tvalid_0's auc: 0.773888\n",
      "[11]\tvalid_0's binary_logloss: 0.457223\tvalid_0's auc: 0.772738\n",
      "[12]\tvalid_0's binary_logloss: 0.452691\tvalid_0's auc: 0.772988\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[1]\tvalid_0's binary_logloss: 0.639997\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598647\tvalid_0's auc: 0.768051\n",
      "[3]\tvalid_0's binary_logloss: 0.56668\tvalid_0's auc: 0.769974\n",
      "[4]\tvalid_0's binary_logloss: 0.541144\tvalid_0's auc: 0.770928\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[6]\tvalid_0's binary_logloss: 0.504237\tvalid_0's auc: 0.771318\n",
      "[7]\tvalid_0's binary_logloss: 0.490907\tvalid_0's auc: 0.770627\n",
      "[8]\tvalid_0's binary_logloss: 0.48037\tvalid_0's auc: 0.769352\n",
      "[9]\tvalid_0's binary_logloss: 0.471982\tvalid_0's auc: 0.768651\n",
      "[10]\tvalid_0's binary_logloss: 0.464704\tvalid_0's auc: 0.769834\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[1]\tvalid_0's binary_logloss: 0.639605\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598235\tvalid_0's auc: 0.770893\n",
      "[3]\tvalid_0's binary_logloss: 0.566282\tvalid_0's auc: 0.772481\n",
      "[4]\tvalid_0's binary_logloss: 0.540521\tvalid_0's auc: 0.774474\n",
      "[5]\tvalid_0's binary_logloss: 0.51982\tvalid_0's auc: 0.775178\n",
      "[6]\tvalid_0's binary_logloss: 0.503527\tvalid_0's auc: 0.774708\n",
      "[7]\tvalid_0's binary_logloss: 0.490433\tvalid_0's auc: 0.77386\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[9]\tvalid_0's binary_logloss: 0.47085\tvalid_0's auc: 0.774124\n",
      "[10]\tvalid_0's binary_logloss: 0.463677\tvalid_0's auc: 0.774711\n",
      "[11]\tvalid_0's binary_logloss: 0.458534\tvalid_0's auc: 0.773313\n",
      "[12]\tvalid_0's binary_logloss: 0.45358\tvalid_0's auc: 0.773877\n",
      "[13]\tvalid_0's binary_logloss: 0.449739\tvalid_0's auc: 0.773154\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[1]\tvalid_0's binary_logloss: 0.639924\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598752\tvalid_0's auc: 0.766736\n",
      "[3]\tvalid_0's binary_logloss: 0.566652\tvalid_0's auc: 0.768039\n",
      "[4]\tvalid_0's binary_logloss: 0.540753\tvalid_0's auc: 0.771686\n",
      "[5]\tvalid_0's binary_logloss: 0.520077\tvalid_0's auc: 0.772744\n",
      "[6]\tvalid_0's binary_logloss: 0.503503\tvalid_0's auc: 0.774082\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[8]\tvalid_0's binary_logloss: 0.4795\tvalid_0's auc: 0.77261\n",
      "[9]\tvalid_0's binary_logloss: 0.470271\tvalid_0's auc: 0.773885\n",
      "[10]\tvalid_0's binary_logloss: 0.463132\tvalid_0's auc: 0.773888\n",
      "[11]\tvalid_0's binary_logloss: 0.457223\tvalid_0's auc: 0.772738\n",
      "[12]\tvalid_0's binary_logloss: 0.452691\tvalid_0's auc: 0.772988\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[1]\tvalid_0's binary_logloss: 0.639997\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598647\tvalid_0's auc: 0.768051\n",
      "[3]\tvalid_0's binary_logloss: 0.56668\tvalid_0's auc: 0.769974\n",
      "[4]\tvalid_0's binary_logloss: 0.541144\tvalid_0's auc: 0.770928\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[6]\tvalid_0's binary_logloss: 0.504237\tvalid_0's auc: 0.771318\n",
      "[7]\tvalid_0's binary_logloss: 0.490907\tvalid_0's auc: 0.770627\n",
      "[8]\tvalid_0's binary_logloss: 0.48037\tvalid_0's auc: 0.769352\n",
      "[9]\tvalid_0's binary_logloss: 0.471982\tvalid_0's auc: 0.768651\n",
      "[10]\tvalid_0's binary_logloss: 0.464704\tvalid_0's auc: 0.769834\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[1]\tvalid_0's binary_logloss: 0.639605\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598235\tvalid_0's auc: 0.770893\n",
      "[3]\tvalid_0's binary_logloss: 0.566282\tvalid_0's auc: 0.772481\n",
      "[4]\tvalid_0's binary_logloss: 0.540521\tvalid_0's auc: 0.774474\n",
      "[5]\tvalid_0's binary_logloss: 0.51982\tvalid_0's auc: 0.775178\n",
      "[6]\tvalid_0's binary_logloss: 0.503527\tvalid_0's auc: 0.774708\n",
      "[7]\tvalid_0's binary_logloss: 0.490433\tvalid_0's auc: 0.77386\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[9]\tvalid_0's binary_logloss: 0.47085\tvalid_0's auc: 0.774124\n",
      "[10]\tvalid_0's binary_logloss: 0.463677\tvalid_0's auc: 0.774711\n",
      "[11]\tvalid_0's binary_logloss: 0.458534\tvalid_0's auc: 0.773313\n",
      "[12]\tvalid_0's binary_logloss: 0.45358\tvalid_0's auc: 0.773877\n",
      "[13]\tvalid_0's binary_logloss: 0.449739\tvalid_0's auc: 0.773154\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[1]\tvalid_0's binary_logloss: 0.639924\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598752\tvalid_0's auc: 0.766736\n",
      "[3]\tvalid_0's binary_logloss: 0.566652\tvalid_0's auc: 0.768039\n",
      "[4]\tvalid_0's binary_logloss: 0.540753\tvalid_0's auc: 0.771686\n",
      "[5]\tvalid_0's binary_logloss: 0.520077\tvalid_0's auc: 0.772744\n",
      "[6]\tvalid_0's binary_logloss: 0.503503\tvalid_0's auc: 0.774082\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n",
      "[8]\tvalid_0's binary_logloss: 0.4795\tvalid_0's auc: 0.77261\n",
      "[9]\tvalid_0's binary_logloss: 0.470271\tvalid_0's auc: 0.773885\n",
      "[10]\tvalid_0's binary_logloss: 0.463132\tvalid_0's auc: 0.773888\n",
      "[11]\tvalid_0's binary_logloss: 0.457223\tvalid_0's auc: 0.772738\n",
      "[12]\tvalid_0's binary_logloss: 0.452691\tvalid_0's auc: 0.772988\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.489934\tvalid_0's auc: 0.774086\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.639997\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598647\tvalid_0's auc: 0.768051\n",
      "[3]\tvalid_0's binary_logloss: 0.56668\tvalid_0's auc: 0.769974\n",
      "[4]\tvalid_0's binary_logloss: 0.541144\tvalid_0's auc: 0.770928\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[6]\tvalid_0's binary_logloss: 0.504237\tvalid_0's auc: 0.771318\n",
      "[7]\tvalid_0's binary_logloss: 0.490907\tvalid_0's auc: 0.770627\n",
      "[8]\tvalid_0's binary_logloss: 0.48037\tvalid_0's auc: 0.769352\n",
      "[9]\tvalid_0's binary_logloss: 0.471982\tvalid_0's auc: 0.768651\n",
      "[10]\tvalid_0's binary_logloss: 0.464704\tvalid_0's auc: 0.769834\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.520396\tvalid_0's auc: 0.771398\n",
      "[1]\tvalid_0's binary_logloss: 0.639605\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.598235\tvalid_0's auc: 0.770893\n",
      "[3]\tvalid_0's binary_logloss: 0.566282\tvalid_0's auc: 0.772481\n",
      "[4]\tvalid_0's binary_logloss: 0.540521\tvalid_0's auc: 0.774474\n",
      "[5]\tvalid_0's binary_logloss: 0.51982\tvalid_0's auc: 0.775178\n",
      "[6]\tvalid_0's binary_logloss: 0.503527\tvalid_0's auc: 0.774708\n",
      "[7]\tvalid_0's binary_logloss: 0.490433\tvalid_0's auc: 0.77386\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[9]\tvalid_0's binary_logloss: 0.47085\tvalid_0's auc: 0.774124\n",
      "[10]\tvalid_0's binary_logloss: 0.463677\tvalid_0's auc: 0.774711\n",
      "[11]\tvalid_0's binary_logloss: 0.458534\tvalid_0's auc: 0.773313\n",
      "[12]\tvalid_0's binary_logloss: 0.45358\tvalid_0's auc: 0.773877\n",
      "[13]\tvalid_0's binary_logloss: 0.449739\tvalid_0's auc: 0.773154\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's binary_logloss: 0.479204\tvalid_0's auc: 0.77598\n",
      "[1]\tvalid_0's binary_logloss: 0.63015\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583691\tvalid_0's auc: 0.769051\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[4]\tvalid_0's binary_logloss: 0.52284\tvalid_0's auc: 0.772433\n",
      "[5]\tvalid_0's binary_logloss: 0.502628\tvalid_0's auc: 0.773269\n",
      "[6]\tvalid_0's binary_logloss: 0.487527\tvalid_0's auc: 0.771441\n",
      "[7]\tvalid_0's binary_logloss: 0.475343\tvalid_0's auc: 0.771977\n",
      "[8]\tvalid_0's binary_logloss: 0.465871\tvalid_0's auc: 0.771464\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[1]\tvalid_0's binary_logloss: 0.630238\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583809\tvalid_0's auc: 0.767986\n",
      "[3]\tvalid_0's binary_logloss: 0.549496\tvalid_0's auc: 0.770526\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[5]\tvalid_0's binary_logloss: 0.50331\tvalid_0's auc: 0.770445\n",
      "[6]\tvalid_0's binary_logloss: 0.487805\tvalid_0's auc: 0.769169\n",
      "[7]\tvalid_0's binary_logloss: 0.476158\tvalid_0's auc: 0.767893\n",
      "[8]\tvalid_0's binary_logloss: 0.466896\tvalid_0's auc: 0.768205\n",
      "[9]\tvalid_0's binary_logloss: 0.459388\tvalid_0's auc: 0.770344\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[1]\tvalid_0's binary_logloss: 0.629761\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583254\tvalid_0's auc: 0.770839\n",
      "[3]\tvalid_0's binary_logloss: 0.54917\tvalid_0's auc: 0.772405\n",
      "[4]\tvalid_0's binary_logloss: 0.522361\tvalid_0's auc: 0.774326\n",
      "[5]\tvalid_0's binary_logloss: 0.502763\tvalid_0's auc: 0.773486\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[7]\tvalid_0's binary_logloss: 0.47501\tvalid_0's auc: 0.77426\n",
      "[8]\tvalid_0's binary_logloss: 0.465609\tvalid_0's auc: 0.773445\n",
      "[9]\tvalid_0's binary_logloss: 0.45834\tvalid_0's auc: 0.774455\n",
      "[10]\tvalid_0's binary_logloss: 0.452445\tvalid_0's auc: 0.773638\n",
      "[11]\tvalid_0's binary_logloss: 0.448374\tvalid_0's auc: 0.773321\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[1]\tvalid_0's binary_logloss: 0.63015\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583691\tvalid_0's auc: 0.769051\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[4]\tvalid_0's binary_logloss: 0.52284\tvalid_0's auc: 0.772433\n",
      "[5]\tvalid_0's binary_logloss: 0.502628\tvalid_0's auc: 0.773269\n",
      "[6]\tvalid_0's binary_logloss: 0.487527\tvalid_0's auc: 0.771441\n",
      "[7]\tvalid_0's binary_logloss: 0.475343\tvalid_0's auc: 0.771977\n",
      "[8]\tvalid_0's binary_logloss: 0.465871\tvalid_0's auc: 0.771464\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[1]\tvalid_0's binary_logloss: 0.630238\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583809\tvalid_0's auc: 0.767986\n",
      "[3]\tvalid_0's binary_logloss: 0.549496\tvalid_0's auc: 0.770526\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[5]\tvalid_0's binary_logloss: 0.50331\tvalid_0's auc: 0.770445\n",
      "[6]\tvalid_0's binary_logloss: 0.487805\tvalid_0's auc: 0.769169\n",
      "[7]\tvalid_0's binary_logloss: 0.476158\tvalid_0's auc: 0.767893\n",
      "[8]\tvalid_0's binary_logloss: 0.466896\tvalid_0's auc: 0.768205\n",
      "[9]\tvalid_0's binary_logloss: 0.459388\tvalid_0's auc: 0.770344\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[1]\tvalid_0's binary_logloss: 0.629761\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583254\tvalid_0's auc: 0.770839\n",
      "[3]\tvalid_0's binary_logloss: 0.54917\tvalid_0's auc: 0.772405\n",
      "[4]\tvalid_0's binary_logloss: 0.522361\tvalid_0's auc: 0.774326\n",
      "[5]\tvalid_0's binary_logloss: 0.502763\tvalid_0's auc: 0.773486\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[7]\tvalid_0's binary_logloss: 0.47501\tvalid_0's auc: 0.77426\n",
      "[8]\tvalid_0's binary_logloss: 0.465609\tvalid_0's auc: 0.773445\n",
      "[9]\tvalid_0's binary_logloss: 0.45834\tvalid_0's auc: 0.774455\n",
      "[10]\tvalid_0's binary_logloss: 0.452445\tvalid_0's auc: 0.773638\n",
      "[11]\tvalid_0's binary_logloss: 0.448374\tvalid_0's auc: 0.773321\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[1]\tvalid_0's binary_logloss: 0.63015\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583691\tvalid_0's auc: 0.769051\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[4]\tvalid_0's binary_logloss: 0.52284\tvalid_0's auc: 0.772433\n",
      "[5]\tvalid_0's binary_logloss: 0.502628\tvalid_0's auc: 0.773269\n",
      "[6]\tvalid_0's binary_logloss: 0.487527\tvalid_0's auc: 0.771441\n",
      "[7]\tvalid_0's binary_logloss: 0.475343\tvalid_0's auc: 0.771977\n",
      "[8]\tvalid_0's binary_logloss: 0.465871\tvalid_0's auc: 0.771464\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[1]\tvalid_0's binary_logloss: 0.630238\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583809\tvalid_0's auc: 0.767986\n",
      "[3]\tvalid_0's binary_logloss: 0.549496\tvalid_0's auc: 0.770526\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[5]\tvalid_0's binary_logloss: 0.50331\tvalid_0's auc: 0.770445\n",
      "[6]\tvalid_0's binary_logloss: 0.487805\tvalid_0's auc: 0.769169\n",
      "[7]\tvalid_0's binary_logloss: 0.476158\tvalid_0's auc: 0.767893\n",
      "[8]\tvalid_0's binary_logloss: 0.466896\tvalid_0's auc: 0.768205\n",
      "[9]\tvalid_0's binary_logloss: 0.459388\tvalid_0's auc: 0.770344\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[1]\tvalid_0's binary_logloss: 0.629761\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583254\tvalid_0's auc: 0.770839\n",
      "[3]\tvalid_0's binary_logloss: 0.54917\tvalid_0's auc: 0.772405\n",
      "[4]\tvalid_0's binary_logloss: 0.522361\tvalid_0's auc: 0.774326\n",
      "[5]\tvalid_0's binary_logloss: 0.502763\tvalid_0's auc: 0.773486\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[7]\tvalid_0's binary_logloss: 0.47501\tvalid_0's auc: 0.77426\n",
      "[8]\tvalid_0's binary_logloss: 0.465609\tvalid_0's auc: 0.773445\n",
      "[9]\tvalid_0's binary_logloss: 0.45834\tvalid_0's auc: 0.774455\n",
      "[10]\tvalid_0's binary_logloss: 0.452445\tvalid_0's auc: 0.773638\n",
      "[11]\tvalid_0's binary_logloss: 0.448374\tvalid_0's auc: 0.773321\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[1]\tvalid_0's binary_logloss: 0.63015\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583691\tvalid_0's auc: 0.769051\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[4]\tvalid_0's binary_logloss: 0.52284\tvalid_0's auc: 0.772433\n",
      "[5]\tvalid_0's binary_logloss: 0.502628\tvalid_0's auc: 0.773269\n",
      "[6]\tvalid_0's binary_logloss: 0.487527\tvalid_0's auc: 0.771441\n",
      "[7]\tvalid_0's binary_logloss: 0.475343\tvalid_0's auc: 0.771977\n",
      "[8]\tvalid_0's binary_logloss: 0.465871\tvalid_0's auc: 0.771464\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.630238\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583809\tvalid_0's auc: 0.767986\n",
      "[3]\tvalid_0's binary_logloss: 0.549496\tvalid_0's auc: 0.770526\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[5]\tvalid_0's binary_logloss: 0.50331\tvalid_0's auc: 0.770445\n",
      "[6]\tvalid_0's binary_logloss: 0.487805\tvalid_0's auc: 0.769169\n",
      "[7]\tvalid_0's binary_logloss: 0.476158\tvalid_0's auc: 0.767893\n",
      "[8]\tvalid_0's binary_logloss: 0.466896\tvalid_0's auc: 0.768205\n",
      "[9]\tvalid_0's binary_logloss: 0.459388\tvalid_0's auc: 0.770344\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[1]\tvalid_0's binary_logloss: 0.629761\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583254\tvalid_0's auc: 0.770839\n",
      "[3]\tvalid_0's binary_logloss: 0.54917\tvalid_0's auc: 0.772405\n",
      "[4]\tvalid_0's binary_logloss: 0.522361\tvalid_0's auc: 0.774326\n",
      "[5]\tvalid_0's binary_logloss: 0.502763\tvalid_0's auc: 0.773486\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[7]\tvalid_0's binary_logloss: 0.47501\tvalid_0's auc: 0.77426\n",
      "[8]\tvalid_0's binary_logloss: 0.465609\tvalid_0's auc: 0.773445\n",
      "[9]\tvalid_0's binary_logloss: 0.45834\tvalid_0's auc: 0.774455\n",
      "[10]\tvalid_0's binary_logloss: 0.452445\tvalid_0's auc: 0.773638\n",
      "[11]\tvalid_0's binary_logloss: 0.448374\tvalid_0's auc: 0.773321\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[1]\tvalid_0's binary_logloss: 0.63015\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583691\tvalid_0's auc: 0.769051\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[4]\tvalid_0's binary_logloss: 0.52284\tvalid_0's auc: 0.772433\n",
      "[5]\tvalid_0's binary_logloss: 0.502628\tvalid_0's auc: 0.773269\n",
      "[6]\tvalid_0's binary_logloss: 0.487527\tvalid_0's auc: 0.771441\n",
      "[7]\tvalid_0's binary_logloss: 0.475343\tvalid_0's auc: 0.771977\n",
      "[8]\tvalid_0's binary_logloss: 0.465871\tvalid_0's auc: 0.771464\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[1]\tvalid_0's binary_logloss: 0.630238\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583809\tvalid_0's auc: 0.767986\n",
      "[3]\tvalid_0's binary_logloss: 0.549496\tvalid_0's auc: 0.770526\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[5]\tvalid_0's binary_logloss: 0.50331\tvalid_0's auc: 0.770445\n",
      "[6]\tvalid_0's binary_logloss: 0.487805\tvalid_0's auc: 0.769169\n",
      "[7]\tvalid_0's binary_logloss: 0.476158\tvalid_0's auc: 0.767893\n",
      "[8]\tvalid_0's binary_logloss: 0.466896\tvalid_0's auc: 0.768205\n",
      "[9]\tvalid_0's binary_logloss: 0.459388\tvalid_0's auc: 0.770344\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[1]\tvalid_0's binary_logloss: 0.629761\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583254\tvalid_0's auc: 0.770839\n",
      "[3]\tvalid_0's binary_logloss: 0.54917\tvalid_0's auc: 0.772405\n",
      "[4]\tvalid_0's binary_logloss: 0.522361\tvalid_0's auc: 0.774326\n",
      "[5]\tvalid_0's binary_logloss: 0.502763\tvalid_0's auc: 0.773486\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[7]\tvalid_0's binary_logloss: 0.47501\tvalid_0's auc: 0.77426\n",
      "[8]\tvalid_0's binary_logloss: 0.465609\tvalid_0's auc: 0.773445\n",
      "[9]\tvalid_0's binary_logloss: 0.45834\tvalid_0's auc: 0.774455\n",
      "[10]\tvalid_0's binary_logloss: 0.452445\tvalid_0's auc: 0.773638\n",
      "[11]\tvalid_0's binary_logloss: 0.448374\tvalid_0's auc: 0.773321\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[1]\tvalid_0's binary_logloss: 0.63015\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583691\tvalid_0's auc: 0.769051\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[4]\tvalid_0's binary_logloss: 0.52284\tvalid_0's auc: 0.772433\n",
      "[5]\tvalid_0's binary_logloss: 0.502628\tvalid_0's auc: 0.773269\n",
      "[6]\tvalid_0's binary_logloss: 0.487527\tvalid_0's auc: 0.771441\n",
      "[7]\tvalid_0's binary_logloss: 0.475343\tvalid_0's auc: 0.771977\n",
      "[8]\tvalid_0's binary_logloss: 0.465871\tvalid_0's auc: 0.771464\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[1]\tvalid_0's binary_logloss: 0.630238\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583809\tvalid_0's auc: 0.767986\n",
      "[3]\tvalid_0's binary_logloss: 0.549496\tvalid_0's auc: 0.770526\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[5]\tvalid_0's binary_logloss: 0.50331\tvalid_0's auc: 0.770445\n",
      "[6]\tvalid_0's binary_logloss: 0.487805\tvalid_0's auc: 0.769169\n",
      "[7]\tvalid_0's binary_logloss: 0.476158\tvalid_0's auc: 0.767893\n",
      "[8]\tvalid_0's binary_logloss: 0.466896\tvalid_0's auc: 0.768205\n",
      "[9]\tvalid_0's binary_logloss: 0.459388\tvalid_0's auc: 0.770344\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[1]\tvalid_0's binary_logloss: 0.629761\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583254\tvalid_0's auc: 0.770839\n",
      "[3]\tvalid_0's binary_logloss: 0.54917\tvalid_0's auc: 0.772405\n",
      "[4]\tvalid_0's binary_logloss: 0.522361\tvalid_0's auc: 0.774326\n",
      "[5]\tvalid_0's binary_logloss: 0.502763\tvalid_0's auc: 0.773486\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[7]\tvalid_0's binary_logloss: 0.47501\tvalid_0's auc: 0.77426\n",
      "[8]\tvalid_0's binary_logloss: 0.465609\tvalid_0's auc: 0.773445\n",
      "[9]\tvalid_0's binary_logloss: 0.45834\tvalid_0's auc: 0.774455\n",
      "[10]\tvalid_0's binary_logloss: 0.452445\tvalid_0's auc: 0.773638\n",
      "[11]\tvalid_0's binary_logloss: 0.448374\tvalid_0's auc: 0.773321\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[1]\tvalid_0's binary_logloss: 0.63015\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583691\tvalid_0's auc: 0.769051\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[4]\tvalid_0's binary_logloss: 0.52284\tvalid_0's auc: 0.772433\n",
      "[5]\tvalid_0's binary_logloss: 0.502628\tvalid_0's auc: 0.773269\n",
      "[6]\tvalid_0's binary_logloss: 0.487527\tvalid_0's auc: 0.771441\n",
      "[7]\tvalid_0's binary_logloss: 0.475343\tvalid_0's auc: 0.771977\n",
      "[8]\tvalid_0's binary_logloss: 0.465871\tvalid_0's auc: 0.771464\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[1]\tvalid_0's binary_logloss: 0.630238\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583809\tvalid_0's auc: 0.767986\n",
      "[3]\tvalid_0's binary_logloss: 0.549496\tvalid_0's auc: 0.770526\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[5]\tvalid_0's binary_logloss: 0.50331\tvalid_0's auc: 0.770445\n",
      "[6]\tvalid_0's binary_logloss: 0.487805\tvalid_0's auc: 0.769169\n",
      "[7]\tvalid_0's binary_logloss: 0.476158\tvalid_0's auc: 0.767893\n",
      "[8]\tvalid_0's binary_logloss: 0.466896\tvalid_0's auc: 0.768205\n",
      "[9]\tvalid_0's binary_logloss: 0.459388\tvalid_0's auc: 0.770344\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[1]\tvalid_0's binary_logloss: 0.629761\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583254\tvalid_0's auc: 0.770839\n",
      "[3]\tvalid_0's binary_logloss: 0.54917\tvalid_0's auc: 0.772405\n",
      "[4]\tvalid_0's binary_logloss: 0.522361\tvalid_0's auc: 0.774326\n",
      "[5]\tvalid_0's binary_logloss: 0.502763\tvalid_0's auc: 0.773486\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[7]\tvalid_0's binary_logloss: 0.47501\tvalid_0's auc: 0.77426\n",
      "[8]\tvalid_0's binary_logloss: 0.465609\tvalid_0's auc: 0.773445\n",
      "[9]\tvalid_0's binary_logloss: 0.45834\tvalid_0's auc: 0.774455\n",
      "[10]\tvalid_0's binary_logloss: 0.452445\tvalid_0's auc: 0.773638\n",
      "[11]\tvalid_0's binary_logloss: 0.448374\tvalid_0's auc: 0.773321\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[1]\tvalid_0's binary_logloss: 0.63015\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583691\tvalid_0's auc: 0.769051\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n",
      "[4]\tvalid_0's binary_logloss: 0.52284\tvalid_0's auc: 0.772433\n",
      "[5]\tvalid_0's binary_logloss: 0.502628\tvalid_0's auc: 0.773269\n",
      "[6]\tvalid_0's binary_logloss: 0.487527\tvalid_0's auc: 0.771441\n",
      "[7]\tvalid_0's binary_logloss: 0.475343\tvalid_0's auc: 0.771977\n",
      "[8]\tvalid_0's binary_logloss: 0.465871\tvalid_0's auc: 0.771464\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.549044\tvalid_0's auc: 0.77421\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.630238\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583809\tvalid_0's auc: 0.767986\n",
      "[3]\tvalid_0's binary_logloss: 0.549496\tvalid_0's auc: 0.770526\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[5]\tvalid_0's binary_logloss: 0.50331\tvalid_0's auc: 0.770445\n",
      "[6]\tvalid_0's binary_logloss: 0.487805\tvalid_0's auc: 0.769169\n",
      "[7]\tvalid_0's binary_logloss: 0.476158\tvalid_0's auc: 0.767893\n",
      "[8]\tvalid_0's binary_logloss: 0.466896\tvalid_0's auc: 0.768205\n",
      "[9]\tvalid_0's binary_logloss: 0.459388\tvalid_0's auc: 0.770344\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.523184\tvalid_0's auc: 0.771226\n",
      "[1]\tvalid_0's binary_logloss: 0.629761\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.583254\tvalid_0's auc: 0.770839\n",
      "[3]\tvalid_0's binary_logloss: 0.54917\tvalid_0's auc: 0.772405\n",
      "[4]\tvalid_0's binary_logloss: 0.522361\tvalid_0's auc: 0.774326\n",
      "[5]\tvalid_0's binary_logloss: 0.502763\tvalid_0's auc: 0.773486\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[7]\tvalid_0's binary_logloss: 0.47501\tvalid_0's auc: 0.77426\n",
      "[8]\tvalid_0's binary_logloss: 0.465609\tvalid_0's auc: 0.773445\n",
      "[9]\tvalid_0's binary_logloss: 0.45834\tvalid_0's auc: 0.774455\n",
      "[10]\tvalid_0's binary_logloss: 0.452445\tvalid_0's auc: 0.773638\n",
      "[11]\tvalid_0's binary_logloss: 0.448374\tvalid_0's auc: 0.773321\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.486832\tvalid_0's auc: 0.774502\n",
      "[1]\tvalid_0's binary_logloss: 0.620663\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570317\tvalid_0's auc: 0.767494\n",
      "[3]\tvalid_0's binary_logloss: 0.534293\tvalid_0's auc: 0.769038\n",
      "[4]\tvalid_0's binary_logloss: 0.508294\tvalid_0's auc: 0.771713\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[6]\tvalid_0's binary_logloss: 0.474719\tvalid_0's auc: 0.772056\n",
      "[7]\tvalid_0's binary_logloss: 0.463578\tvalid_0's auc: 0.773318\n",
      "[8]\tvalid_0's binary_logloss: 0.455916\tvalid_0's auc: 0.773168\n",
      "[9]\tvalid_0's binary_logloss: 0.449915\tvalid_0's auc: 0.773491\n",
      "[10]\tvalid_0's binary_logloss: 0.44554\tvalid_0's auc: 0.773798\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[1]\tvalid_0's binary_logloss: 0.620767\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570096\tvalid_0's auc: 0.768441\n",
      "[3]\tvalid_0's binary_logloss: 0.533829\tvalid_0's auc: 0.772067\n",
      "[4]\tvalid_0's binary_logloss: 0.507845\tvalid_0's auc: 0.771469\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[6]\tvalid_0's binary_logloss: 0.474513\tvalid_0's auc: 0.771085\n",
      "[7]\tvalid_0's binary_logloss: 0.463801\tvalid_0's auc: 0.770495\n",
      "[8]\tvalid_0's binary_logloss: 0.45602\tvalid_0's auc: 0.770549\n",
      "[9]\tvalid_0's binary_logloss: 0.450541\tvalid_0's auc: 0.770545\n",
      "[10]\tvalid_0's binary_logloss: 0.446428\tvalid_0's auc: 0.770508\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[1]\tvalid_0's binary_logloss: 0.620202\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.56942\tvalid_0's auc: 0.770509\n",
      "[3]\tvalid_0's binary_logloss: 0.534157\tvalid_0's auc: 0.771202\n",
      "[4]\tvalid_0's binary_logloss: 0.507424\tvalid_0's auc: 0.773188\n",
      "[5]\tvalid_0's binary_logloss: 0.488424\tvalid_0's auc: 0.772624\n",
      "[6]\tvalid_0's binary_logloss: 0.474588\tvalid_0's auc: 0.772814\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[8]\tvalid_0's binary_logloss: 0.45589\tvalid_0's auc: 0.774939\n",
      "[9]\tvalid_0's binary_logloss: 0.450026\tvalid_0's auc: 0.774092\n",
      "[10]\tvalid_0's binary_logloss: 0.445935\tvalid_0's auc: 0.773524\n",
      "[11]\tvalid_0's binary_logloss: 0.442718\tvalid_0's auc: 0.773262\n",
      "[12]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.771102\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[1]\tvalid_0's binary_logloss: 0.620663\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570317\tvalid_0's auc: 0.767494\n",
      "[3]\tvalid_0's binary_logloss: 0.534293\tvalid_0's auc: 0.769038\n",
      "[4]\tvalid_0's binary_logloss: 0.508294\tvalid_0's auc: 0.771713\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[6]\tvalid_0's binary_logloss: 0.474719\tvalid_0's auc: 0.772056\n",
      "[7]\tvalid_0's binary_logloss: 0.463578\tvalid_0's auc: 0.773318\n",
      "[8]\tvalid_0's binary_logloss: 0.455916\tvalid_0's auc: 0.773168\n",
      "[9]\tvalid_0's binary_logloss: 0.449915\tvalid_0's auc: 0.773491\n",
      "[10]\tvalid_0's binary_logloss: 0.44554\tvalid_0's auc: 0.773798\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[1]\tvalid_0's binary_logloss: 0.620767\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570096\tvalid_0's auc: 0.768441\n",
      "[3]\tvalid_0's binary_logloss: 0.533829\tvalid_0's auc: 0.772067\n",
      "[4]\tvalid_0's binary_logloss: 0.507845\tvalid_0's auc: 0.771469\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[6]\tvalid_0's binary_logloss: 0.474513\tvalid_0's auc: 0.771085\n",
      "[7]\tvalid_0's binary_logloss: 0.463801\tvalid_0's auc: 0.770495\n",
      "[8]\tvalid_0's binary_logloss: 0.45602\tvalid_0's auc: 0.770549\n",
      "[9]\tvalid_0's binary_logloss: 0.450541\tvalid_0's auc: 0.770545\n",
      "[10]\tvalid_0's binary_logloss: 0.446428\tvalid_0's auc: 0.770508\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[1]\tvalid_0's binary_logloss: 0.620202\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.56942\tvalid_0's auc: 0.770509\n",
      "[3]\tvalid_0's binary_logloss: 0.534157\tvalid_0's auc: 0.771202\n",
      "[4]\tvalid_0's binary_logloss: 0.507424\tvalid_0's auc: 0.773188\n",
      "[5]\tvalid_0's binary_logloss: 0.488424\tvalid_0's auc: 0.772624\n",
      "[6]\tvalid_0's binary_logloss: 0.474588\tvalid_0's auc: 0.772814\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[8]\tvalid_0's binary_logloss: 0.45589\tvalid_0's auc: 0.774939\n",
      "[9]\tvalid_0's binary_logloss: 0.450026\tvalid_0's auc: 0.774092\n",
      "[10]\tvalid_0's binary_logloss: 0.445935\tvalid_0's auc: 0.773524\n",
      "[11]\tvalid_0's binary_logloss: 0.442718\tvalid_0's auc: 0.773262\n",
      "[12]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.771102\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[1]\tvalid_0's binary_logloss: 0.620663\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570317\tvalid_0's auc: 0.767494\n",
      "[3]\tvalid_0's binary_logloss: 0.534293\tvalid_0's auc: 0.769038\n",
      "[4]\tvalid_0's binary_logloss: 0.508294\tvalid_0's auc: 0.771713\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[6]\tvalid_0's binary_logloss: 0.474719\tvalid_0's auc: 0.772056\n",
      "[7]\tvalid_0's binary_logloss: 0.463578\tvalid_0's auc: 0.773318\n",
      "[8]\tvalid_0's binary_logloss: 0.455916\tvalid_0's auc: 0.773168\n",
      "[9]\tvalid_0's binary_logloss: 0.449915\tvalid_0's auc: 0.773491\n",
      "[10]\tvalid_0's binary_logloss: 0.44554\tvalid_0's auc: 0.773798\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[1]\tvalid_0's binary_logloss: 0.620767\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570096\tvalid_0's auc: 0.768441\n",
      "[3]\tvalid_0's binary_logloss: 0.533829\tvalid_0's auc: 0.772067\n",
      "[4]\tvalid_0's binary_logloss: 0.507845\tvalid_0's auc: 0.771469\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[6]\tvalid_0's binary_logloss: 0.474513\tvalid_0's auc: 0.771085\n",
      "[7]\tvalid_0's binary_logloss: 0.463801\tvalid_0's auc: 0.770495\n",
      "[8]\tvalid_0's binary_logloss: 0.45602\tvalid_0's auc: 0.770549\n",
      "[9]\tvalid_0's binary_logloss: 0.450541\tvalid_0's auc: 0.770545\n",
      "[10]\tvalid_0's binary_logloss: 0.446428\tvalid_0's auc: 0.770508\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[1]\tvalid_0's binary_logloss: 0.620202\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.56942\tvalid_0's auc: 0.770509\n",
      "[3]\tvalid_0's binary_logloss: 0.534157\tvalid_0's auc: 0.771202\n",
      "[4]\tvalid_0's binary_logloss: 0.507424\tvalid_0's auc: 0.773188\n",
      "[5]\tvalid_0's binary_logloss: 0.488424\tvalid_0's auc: 0.772624\n",
      "[6]\tvalid_0's binary_logloss: 0.474588\tvalid_0's auc: 0.772814\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[8]\tvalid_0's binary_logloss: 0.45589\tvalid_0's auc: 0.774939\n",
      "[9]\tvalid_0's binary_logloss: 0.450026\tvalid_0's auc: 0.774092\n",
      "[10]\tvalid_0's binary_logloss: 0.445935\tvalid_0's auc: 0.773524\n",
      "[11]\tvalid_0's binary_logloss: 0.442718\tvalid_0's auc: 0.773262\n",
      "[12]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.771102\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[1]\tvalid_0's binary_logloss: 0.620663\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570317\tvalid_0's auc: 0.767494\n",
      "[3]\tvalid_0's binary_logloss: 0.534293\tvalid_0's auc: 0.769038\n",
      "[4]\tvalid_0's binary_logloss: 0.508294\tvalid_0's auc: 0.771713\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[6]\tvalid_0's binary_logloss: 0.474719\tvalid_0's auc: 0.772056\n",
      "[7]\tvalid_0's binary_logloss: 0.463578\tvalid_0's auc: 0.773318\n",
      "[8]\tvalid_0's binary_logloss: 0.455916\tvalid_0's auc: 0.773168\n",
      "[9]\tvalid_0's binary_logloss: 0.449915\tvalid_0's auc: 0.773491\n",
      "[10]\tvalid_0's binary_logloss: 0.44554\tvalid_0's auc: 0.773798\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.620767\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570096\tvalid_0's auc: 0.768441\n",
      "[3]\tvalid_0's binary_logloss: 0.533829\tvalid_0's auc: 0.772067\n",
      "[4]\tvalid_0's binary_logloss: 0.507845\tvalid_0's auc: 0.771469\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[6]\tvalid_0's binary_logloss: 0.474513\tvalid_0's auc: 0.771085\n",
      "[7]\tvalid_0's binary_logloss: 0.463801\tvalid_0's auc: 0.770495\n",
      "[8]\tvalid_0's binary_logloss: 0.45602\tvalid_0's auc: 0.770549\n",
      "[9]\tvalid_0's binary_logloss: 0.450541\tvalid_0's auc: 0.770545\n",
      "[10]\tvalid_0's binary_logloss: 0.446428\tvalid_0's auc: 0.770508\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[1]\tvalid_0's binary_logloss: 0.620202\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.56942\tvalid_0's auc: 0.770509\n",
      "[3]\tvalid_0's binary_logloss: 0.534157\tvalid_0's auc: 0.771202\n",
      "[4]\tvalid_0's binary_logloss: 0.507424\tvalid_0's auc: 0.773188\n",
      "[5]\tvalid_0's binary_logloss: 0.488424\tvalid_0's auc: 0.772624\n",
      "[6]\tvalid_0's binary_logloss: 0.474588\tvalid_0's auc: 0.772814\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[8]\tvalid_0's binary_logloss: 0.45589\tvalid_0's auc: 0.774939\n",
      "[9]\tvalid_0's binary_logloss: 0.450026\tvalid_0's auc: 0.774092\n",
      "[10]\tvalid_0's binary_logloss: 0.445935\tvalid_0's auc: 0.773524\n",
      "[11]\tvalid_0's binary_logloss: 0.442718\tvalid_0's auc: 0.773262\n",
      "[12]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.771102\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[1]\tvalid_0's binary_logloss: 0.620663\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570317\tvalid_0's auc: 0.767494\n",
      "[3]\tvalid_0's binary_logloss: 0.534293\tvalid_0's auc: 0.769038\n",
      "[4]\tvalid_0's binary_logloss: 0.508294\tvalid_0's auc: 0.771713\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[6]\tvalid_0's binary_logloss: 0.474719\tvalid_0's auc: 0.772056\n",
      "[7]\tvalid_0's binary_logloss: 0.463578\tvalid_0's auc: 0.773318\n",
      "[8]\tvalid_0's binary_logloss: 0.455916\tvalid_0's auc: 0.773168\n",
      "[9]\tvalid_0's binary_logloss: 0.449915\tvalid_0's auc: 0.773491\n",
      "[10]\tvalid_0's binary_logloss: 0.44554\tvalid_0's auc: 0.773798\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[1]\tvalid_0's binary_logloss: 0.620767\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570096\tvalid_0's auc: 0.768441\n",
      "[3]\tvalid_0's binary_logloss: 0.533829\tvalid_0's auc: 0.772067\n",
      "[4]\tvalid_0's binary_logloss: 0.507845\tvalid_0's auc: 0.771469\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[6]\tvalid_0's binary_logloss: 0.474513\tvalid_0's auc: 0.771085\n",
      "[7]\tvalid_0's binary_logloss: 0.463801\tvalid_0's auc: 0.770495\n",
      "[8]\tvalid_0's binary_logloss: 0.45602\tvalid_0's auc: 0.770549\n",
      "[9]\tvalid_0's binary_logloss: 0.450541\tvalid_0's auc: 0.770545\n",
      "[10]\tvalid_0's binary_logloss: 0.446428\tvalid_0's auc: 0.770508\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[1]\tvalid_0's binary_logloss: 0.620202\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.56942\tvalid_0's auc: 0.770509\n",
      "[3]\tvalid_0's binary_logloss: 0.534157\tvalid_0's auc: 0.771202\n",
      "[4]\tvalid_0's binary_logloss: 0.507424\tvalid_0's auc: 0.773188\n",
      "[5]\tvalid_0's binary_logloss: 0.488424\tvalid_0's auc: 0.772624\n",
      "[6]\tvalid_0's binary_logloss: 0.474588\tvalid_0's auc: 0.772814\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[8]\tvalid_0's binary_logloss: 0.45589\tvalid_0's auc: 0.774939\n",
      "[9]\tvalid_0's binary_logloss: 0.450026\tvalid_0's auc: 0.774092\n",
      "[10]\tvalid_0's binary_logloss: 0.445935\tvalid_0's auc: 0.773524\n",
      "[11]\tvalid_0's binary_logloss: 0.442718\tvalid_0's auc: 0.773262\n",
      "[12]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.771102\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[1]\tvalid_0's binary_logloss: 0.620663\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570317\tvalid_0's auc: 0.767494\n",
      "[3]\tvalid_0's binary_logloss: 0.534293\tvalid_0's auc: 0.769038\n",
      "[4]\tvalid_0's binary_logloss: 0.508294\tvalid_0's auc: 0.771713\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[6]\tvalid_0's binary_logloss: 0.474719\tvalid_0's auc: 0.772056\n",
      "[7]\tvalid_0's binary_logloss: 0.463578\tvalid_0's auc: 0.773318\n",
      "[8]\tvalid_0's binary_logloss: 0.455916\tvalid_0's auc: 0.773168\n",
      "[9]\tvalid_0's binary_logloss: 0.449915\tvalid_0's auc: 0.773491\n",
      "[10]\tvalid_0's binary_logloss: 0.44554\tvalid_0's auc: 0.773798\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[1]\tvalid_0's binary_logloss: 0.620767\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570096\tvalid_0's auc: 0.768441\n",
      "[3]\tvalid_0's binary_logloss: 0.533829\tvalid_0's auc: 0.772067\n",
      "[4]\tvalid_0's binary_logloss: 0.507845\tvalid_0's auc: 0.771469\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[6]\tvalid_0's binary_logloss: 0.474513\tvalid_0's auc: 0.771085\n",
      "[7]\tvalid_0's binary_logloss: 0.463801\tvalid_0's auc: 0.770495\n",
      "[8]\tvalid_0's binary_logloss: 0.45602\tvalid_0's auc: 0.770549\n",
      "[9]\tvalid_0's binary_logloss: 0.450541\tvalid_0's auc: 0.770545\n",
      "[10]\tvalid_0's binary_logloss: 0.446428\tvalid_0's auc: 0.770508\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[1]\tvalid_0's binary_logloss: 0.620202\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.56942\tvalid_0's auc: 0.770509\n",
      "[3]\tvalid_0's binary_logloss: 0.534157\tvalid_0's auc: 0.771202\n",
      "[4]\tvalid_0's binary_logloss: 0.507424\tvalid_0's auc: 0.773188\n",
      "[5]\tvalid_0's binary_logloss: 0.488424\tvalid_0's auc: 0.772624\n",
      "[6]\tvalid_0's binary_logloss: 0.474588\tvalid_0's auc: 0.772814\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[8]\tvalid_0's binary_logloss: 0.45589\tvalid_0's auc: 0.774939\n",
      "[9]\tvalid_0's binary_logloss: 0.450026\tvalid_0's auc: 0.774092\n",
      "[10]\tvalid_0's binary_logloss: 0.445935\tvalid_0's auc: 0.773524\n",
      "[11]\tvalid_0's binary_logloss: 0.442718\tvalid_0's auc: 0.773262\n",
      "[12]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.771102\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[1]\tvalid_0's binary_logloss: 0.620663\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570317\tvalid_0's auc: 0.767494\n",
      "[3]\tvalid_0's binary_logloss: 0.534293\tvalid_0's auc: 0.769038\n",
      "[4]\tvalid_0's binary_logloss: 0.508294\tvalid_0's auc: 0.771713\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[6]\tvalid_0's binary_logloss: 0.474719\tvalid_0's auc: 0.772056\n",
      "[7]\tvalid_0's binary_logloss: 0.463578\tvalid_0's auc: 0.773318\n",
      "[8]\tvalid_0's binary_logloss: 0.455916\tvalid_0's auc: 0.773168\n",
      "[9]\tvalid_0's binary_logloss: 0.449915\tvalid_0's auc: 0.773491\n",
      "[10]\tvalid_0's binary_logloss: 0.44554\tvalid_0's auc: 0.773798\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[1]\tvalid_0's binary_logloss: 0.620767\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570096\tvalid_0's auc: 0.768441\n",
      "[3]\tvalid_0's binary_logloss: 0.533829\tvalid_0's auc: 0.772067\n",
      "[4]\tvalid_0's binary_logloss: 0.507845\tvalid_0's auc: 0.771469\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[6]\tvalid_0's binary_logloss: 0.474513\tvalid_0's auc: 0.771085\n",
      "[7]\tvalid_0's binary_logloss: 0.463801\tvalid_0's auc: 0.770495\n",
      "[8]\tvalid_0's binary_logloss: 0.45602\tvalid_0's auc: 0.770549\n",
      "[9]\tvalid_0's binary_logloss: 0.450541\tvalid_0's auc: 0.770545\n",
      "[10]\tvalid_0's binary_logloss: 0.446428\tvalid_0's auc: 0.770508\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[1]\tvalid_0's binary_logloss: 0.620202\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.56942\tvalid_0's auc: 0.770509\n",
      "[3]\tvalid_0's binary_logloss: 0.534157\tvalid_0's auc: 0.771202\n",
      "[4]\tvalid_0's binary_logloss: 0.507424\tvalid_0's auc: 0.773188\n",
      "[5]\tvalid_0's binary_logloss: 0.488424\tvalid_0's auc: 0.772624\n",
      "[6]\tvalid_0's binary_logloss: 0.474588\tvalid_0's auc: 0.772814\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[8]\tvalid_0's binary_logloss: 0.45589\tvalid_0's auc: 0.774939\n",
      "[9]\tvalid_0's binary_logloss: 0.450026\tvalid_0's auc: 0.774092\n",
      "[10]\tvalid_0's binary_logloss: 0.445935\tvalid_0's auc: 0.773524\n",
      "[11]\tvalid_0's binary_logloss: 0.442718\tvalid_0's auc: 0.773262\n",
      "[12]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.771102\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.620663\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570317\tvalid_0's auc: 0.767494\n",
      "[3]\tvalid_0's binary_logloss: 0.534293\tvalid_0's auc: 0.769038\n",
      "[4]\tvalid_0's binary_logloss: 0.508294\tvalid_0's auc: 0.771713\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[6]\tvalid_0's binary_logloss: 0.474719\tvalid_0's auc: 0.772056\n",
      "[7]\tvalid_0's binary_logloss: 0.463578\tvalid_0's auc: 0.773318\n",
      "[8]\tvalid_0's binary_logloss: 0.455916\tvalid_0's auc: 0.773168\n",
      "[9]\tvalid_0's binary_logloss: 0.449915\tvalid_0's auc: 0.773491\n",
      "[10]\tvalid_0's binary_logloss: 0.44554\tvalid_0's auc: 0.773798\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.488626\tvalid_0's auc: 0.774199\n",
      "[1]\tvalid_0's binary_logloss: 0.620767\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.570096\tvalid_0's auc: 0.768441\n",
      "[3]\tvalid_0's binary_logloss: 0.533829\tvalid_0's auc: 0.772067\n",
      "[4]\tvalid_0's binary_logloss: 0.507845\tvalid_0's auc: 0.771469\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[6]\tvalid_0's binary_logloss: 0.474513\tvalid_0's auc: 0.771085\n",
      "[7]\tvalid_0's binary_logloss: 0.463801\tvalid_0's auc: 0.770495\n",
      "[8]\tvalid_0's binary_logloss: 0.45602\tvalid_0's auc: 0.770549\n",
      "[9]\tvalid_0's binary_logloss: 0.450541\tvalid_0's auc: 0.770545\n",
      "[10]\tvalid_0's binary_logloss: 0.446428\tvalid_0's auc: 0.770508\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.48866\tvalid_0's auc: 0.772246\n",
      "[1]\tvalid_0's binary_logloss: 0.620202\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.56942\tvalid_0's auc: 0.770509\n",
      "[3]\tvalid_0's binary_logloss: 0.534157\tvalid_0's auc: 0.771202\n",
      "[4]\tvalid_0's binary_logloss: 0.507424\tvalid_0's auc: 0.773188\n",
      "[5]\tvalid_0's binary_logloss: 0.488424\tvalid_0's auc: 0.772624\n",
      "[6]\tvalid_0's binary_logloss: 0.474588\tvalid_0's auc: 0.772814\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[8]\tvalid_0's binary_logloss: 0.45589\tvalid_0's auc: 0.774939\n",
      "[9]\tvalid_0's binary_logloss: 0.450026\tvalid_0's auc: 0.774092\n",
      "[10]\tvalid_0's binary_logloss: 0.445935\tvalid_0's auc: 0.773524\n",
      "[11]\tvalid_0's binary_logloss: 0.442718\tvalid_0's auc: 0.773262\n",
      "[12]\tvalid_0's binary_logloss: 0.441145\tvalid_0's auc: 0.771102\n",
      "Early stopping, best iteration is:\n",
      "[7]\tvalid_0's binary_logloss: 0.463478\tvalid_0's auc: 0.775109\n",
      "[1]\tvalid_0's binary_logloss: 0.611464\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557735\tvalid_0's auc: 0.76943\n",
      "[3]\tvalid_0's binary_logloss: 0.520989\tvalid_0's auc: 0.769779\n",
      "[4]\tvalid_0's binary_logloss: 0.495424\tvalid_0's auc: 0.771953\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[6]\tvalid_0's binary_logloss: 0.464843\tvalid_0's auc: 0.770981\n",
      "[7]\tvalid_0's binary_logloss: 0.455878\tvalid_0's auc: 0.769906\n",
      "[8]\tvalid_0's binary_logloss: 0.449745\tvalid_0's auc: 0.770067\n",
      "[9]\tvalid_0's binary_logloss: 0.4452\tvalid_0's auc: 0.771402\n",
      "[10]\tvalid_0's binary_logloss: 0.442787\tvalid_0's auc: 0.769621\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[1]\tvalid_0's binary_logloss: 0.611582\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557781\tvalid_0's auc: 0.767552\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[4]\tvalid_0's binary_logloss: 0.495612\tvalid_0's auc: 0.768695\n",
      "[5]\tvalid_0's binary_logloss: 0.477462\tvalid_0's auc: 0.768047\n",
      "[6]\tvalid_0's binary_logloss: 0.464938\tvalid_0's auc: 0.766633\n",
      "[7]\tvalid_0's binary_logloss: 0.455599\tvalid_0's auc: 0.769597\n",
      "[8]\tvalid_0's binary_logloss: 0.449157\tvalid_0's auc: 0.769673\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[1]\tvalid_0's binary_logloss: 0.610929\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.556723\tvalid_0's auc: 0.77257\n",
      "[3]\tvalid_0's binary_logloss: 0.521014\tvalid_0's auc: 0.773214\n",
      "[4]\tvalid_0's binary_logloss: 0.495392\tvalid_0's auc: 0.773113\n",
      "[5]\tvalid_0's binary_logloss: 0.477312\tvalid_0's auc: 0.773246\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[7]\tvalid_0's binary_logloss: 0.455974\tvalid_0's auc: 0.773059\n",
      "[8]\tvalid_0's binary_logloss: 0.449487\tvalid_0's auc: 0.771506\n",
      "[9]\tvalid_0's binary_logloss: 0.444703\tvalid_0's auc: 0.771652\n",
      "[10]\tvalid_0's binary_logloss: 0.441448\tvalid_0's auc: 0.771613\n",
      "[11]\tvalid_0's binary_logloss: 0.438751\tvalid_0's auc: 0.77262\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[1]\tvalid_0's binary_logloss: 0.611464\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557735\tvalid_0's auc: 0.76943\n",
      "[3]\tvalid_0's binary_logloss: 0.520989\tvalid_0's auc: 0.769779\n",
      "[4]\tvalid_0's binary_logloss: 0.495424\tvalid_0's auc: 0.771953\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[6]\tvalid_0's binary_logloss: 0.464843\tvalid_0's auc: 0.770981\n",
      "[7]\tvalid_0's binary_logloss: 0.455878\tvalid_0's auc: 0.769906\n",
      "[8]\tvalid_0's binary_logloss: 0.449745\tvalid_0's auc: 0.770067\n",
      "[9]\tvalid_0's binary_logloss: 0.4452\tvalid_0's auc: 0.771402\n",
      "[10]\tvalid_0's binary_logloss: 0.442787\tvalid_0's auc: 0.769621\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[1]\tvalid_0's binary_logloss: 0.611582\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557781\tvalid_0's auc: 0.767552\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[4]\tvalid_0's binary_logloss: 0.495612\tvalid_0's auc: 0.768695\n",
      "[5]\tvalid_0's binary_logloss: 0.477462\tvalid_0's auc: 0.768047\n",
      "[6]\tvalid_0's binary_logloss: 0.464938\tvalid_0's auc: 0.766633\n",
      "[7]\tvalid_0's binary_logloss: 0.455599\tvalid_0's auc: 0.769597\n",
      "[8]\tvalid_0's binary_logloss: 0.449157\tvalid_0's auc: 0.769673\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[1]\tvalid_0's binary_logloss: 0.610929\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.556723\tvalid_0's auc: 0.77257\n",
      "[3]\tvalid_0's binary_logloss: 0.521014\tvalid_0's auc: 0.773214\n",
      "[4]\tvalid_0's binary_logloss: 0.495392\tvalid_0's auc: 0.773113\n",
      "[5]\tvalid_0's binary_logloss: 0.477312\tvalid_0's auc: 0.773246\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[7]\tvalid_0's binary_logloss: 0.455974\tvalid_0's auc: 0.773059\n",
      "[8]\tvalid_0's binary_logloss: 0.449487\tvalid_0's auc: 0.771506\n",
      "[9]\tvalid_0's binary_logloss: 0.444703\tvalid_0's auc: 0.771652\n",
      "[10]\tvalid_0's binary_logloss: 0.441448\tvalid_0's auc: 0.771613\n",
      "[11]\tvalid_0's binary_logloss: 0.438751\tvalid_0's auc: 0.77262\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[1]\tvalid_0's binary_logloss: 0.611464\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557735\tvalid_0's auc: 0.76943\n",
      "[3]\tvalid_0's binary_logloss: 0.520989\tvalid_0's auc: 0.769779\n",
      "[4]\tvalid_0's binary_logloss: 0.495424\tvalid_0's auc: 0.771953\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[6]\tvalid_0's binary_logloss: 0.464843\tvalid_0's auc: 0.770981\n",
      "[7]\tvalid_0's binary_logloss: 0.455878\tvalid_0's auc: 0.769906\n",
      "[8]\tvalid_0's binary_logloss: 0.449745\tvalid_0's auc: 0.770067\n",
      "[9]\tvalid_0's binary_logloss: 0.4452\tvalid_0's auc: 0.771402\n",
      "[10]\tvalid_0's binary_logloss: 0.442787\tvalid_0's auc: 0.769621\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[1]\tvalid_0's binary_logloss: 0.611582\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557781\tvalid_0's auc: 0.767552\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[4]\tvalid_0's binary_logloss: 0.495612\tvalid_0's auc: 0.768695\n",
      "[5]\tvalid_0's binary_logloss: 0.477462\tvalid_0's auc: 0.768047\n",
      "[6]\tvalid_0's binary_logloss: 0.464938\tvalid_0's auc: 0.766633\n",
      "[7]\tvalid_0's binary_logloss: 0.455599\tvalid_0's auc: 0.769597\n",
      "[8]\tvalid_0's binary_logloss: 0.449157\tvalid_0's auc: 0.769673\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[1]\tvalid_0's binary_logloss: 0.610929\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.556723\tvalid_0's auc: 0.77257\n",
      "[3]\tvalid_0's binary_logloss: 0.521014\tvalid_0's auc: 0.773214\n",
      "[4]\tvalid_0's binary_logloss: 0.495392\tvalid_0's auc: 0.773113\n",
      "[5]\tvalid_0's binary_logloss: 0.477312\tvalid_0's auc: 0.773246\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[7]\tvalid_0's binary_logloss: 0.455974\tvalid_0's auc: 0.773059\n",
      "[8]\tvalid_0's binary_logloss: 0.449487\tvalid_0's auc: 0.771506\n",
      "[9]\tvalid_0's binary_logloss: 0.444703\tvalid_0's auc: 0.771652\n",
      "[10]\tvalid_0's binary_logloss: 0.441448\tvalid_0's auc: 0.771613\n",
      "[11]\tvalid_0's binary_logloss: 0.438751\tvalid_0's auc: 0.77262\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[1]\tvalid_0's binary_logloss: 0.611464\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557735\tvalid_0's auc: 0.76943\n",
      "[3]\tvalid_0's binary_logloss: 0.520989\tvalid_0's auc: 0.769779\n",
      "[4]\tvalid_0's binary_logloss: 0.495424\tvalid_0's auc: 0.771953\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[6]\tvalid_0's binary_logloss: 0.464843\tvalid_0's auc: 0.770981\n",
      "[7]\tvalid_0's binary_logloss: 0.455878\tvalid_0's auc: 0.769906\n",
      "[8]\tvalid_0's binary_logloss: 0.449745\tvalid_0's auc: 0.770067\n",
      "[9]\tvalid_0's binary_logloss: 0.4452\tvalid_0's auc: 0.771402\n",
      "[10]\tvalid_0's binary_logloss: 0.442787\tvalid_0's auc: 0.769621\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.611582\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557781\tvalid_0's auc: 0.767552\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[4]\tvalid_0's binary_logloss: 0.495612\tvalid_0's auc: 0.768695\n",
      "[5]\tvalid_0's binary_logloss: 0.477462\tvalid_0's auc: 0.768047\n",
      "[6]\tvalid_0's binary_logloss: 0.464938\tvalid_0's auc: 0.766633\n",
      "[7]\tvalid_0's binary_logloss: 0.455599\tvalid_0's auc: 0.769597\n",
      "[8]\tvalid_0's binary_logloss: 0.449157\tvalid_0's auc: 0.769673\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[1]\tvalid_0's binary_logloss: 0.610929\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.556723\tvalid_0's auc: 0.77257\n",
      "[3]\tvalid_0's binary_logloss: 0.521014\tvalid_0's auc: 0.773214\n",
      "[4]\tvalid_0's binary_logloss: 0.495392\tvalid_0's auc: 0.773113\n",
      "[5]\tvalid_0's binary_logloss: 0.477312\tvalid_0's auc: 0.773246\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[7]\tvalid_0's binary_logloss: 0.455974\tvalid_0's auc: 0.773059\n",
      "[8]\tvalid_0's binary_logloss: 0.449487\tvalid_0's auc: 0.771506\n",
      "[9]\tvalid_0's binary_logloss: 0.444703\tvalid_0's auc: 0.771652\n",
      "[10]\tvalid_0's binary_logloss: 0.441448\tvalid_0's auc: 0.771613\n",
      "[11]\tvalid_0's binary_logloss: 0.438751\tvalid_0's auc: 0.77262\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[1]\tvalid_0's binary_logloss: 0.611464\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557735\tvalid_0's auc: 0.76943\n",
      "[3]\tvalid_0's binary_logloss: 0.520989\tvalid_0's auc: 0.769779\n",
      "[4]\tvalid_0's binary_logloss: 0.495424\tvalid_0's auc: 0.771953\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[6]\tvalid_0's binary_logloss: 0.464843\tvalid_0's auc: 0.770981\n",
      "[7]\tvalid_0's binary_logloss: 0.455878\tvalid_0's auc: 0.769906\n",
      "[8]\tvalid_0's binary_logloss: 0.449745\tvalid_0's auc: 0.770067\n",
      "[9]\tvalid_0's binary_logloss: 0.4452\tvalid_0's auc: 0.771402\n",
      "[10]\tvalid_0's binary_logloss: 0.442787\tvalid_0's auc: 0.769621\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[1]\tvalid_0's binary_logloss: 0.611582\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557781\tvalid_0's auc: 0.767552\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[4]\tvalid_0's binary_logloss: 0.495612\tvalid_0's auc: 0.768695\n",
      "[5]\tvalid_0's binary_logloss: 0.477462\tvalid_0's auc: 0.768047\n",
      "[6]\tvalid_0's binary_logloss: 0.464938\tvalid_0's auc: 0.766633\n",
      "[7]\tvalid_0's binary_logloss: 0.455599\tvalid_0's auc: 0.769597\n",
      "[8]\tvalid_0's binary_logloss: 0.449157\tvalid_0's auc: 0.769673\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[1]\tvalid_0's binary_logloss: 0.610929\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.556723\tvalid_0's auc: 0.77257\n",
      "[3]\tvalid_0's binary_logloss: 0.521014\tvalid_0's auc: 0.773214\n",
      "[4]\tvalid_0's binary_logloss: 0.495392\tvalid_0's auc: 0.773113\n",
      "[5]\tvalid_0's binary_logloss: 0.477312\tvalid_0's auc: 0.773246\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[7]\tvalid_0's binary_logloss: 0.455974\tvalid_0's auc: 0.773059\n",
      "[8]\tvalid_0's binary_logloss: 0.449487\tvalid_0's auc: 0.771506\n",
      "[9]\tvalid_0's binary_logloss: 0.444703\tvalid_0's auc: 0.771652\n",
      "[10]\tvalid_0's binary_logloss: 0.441448\tvalid_0's auc: 0.771613\n",
      "[11]\tvalid_0's binary_logloss: 0.438751\tvalid_0's auc: 0.77262\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[1]\tvalid_0's binary_logloss: 0.611464\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557735\tvalid_0's auc: 0.76943\n",
      "[3]\tvalid_0's binary_logloss: 0.520989\tvalid_0's auc: 0.769779\n",
      "[4]\tvalid_0's binary_logloss: 0.495424\tvalid_0's auc: 0.771953\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[6]\tvalid_0's binary_logloss: 0.464843\tvalid_0's auc: 0.770981\n",
      "[7]\tvalid_0's binary_logloss: 0.455878\tvalid_0's auc: 0.769906\n",
      "[8]\tvalid_0's binary_logloss: 0.449745\tvalid_0's auc: 0.770067\n",
      "[9]\tvalid_0's binary_logloss: 0.4452\tvalid_0's auc: 0.771402\n",
      "[10]\tvalid_0's binary_logloss: 0.442787\tvalid_0's auc: 0.769621\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[1]\tvalid_0's binary_logloss: 0.611582\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557781\tvalid_0's auc: 0.767552\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[4]\tvalid_0's binary_logloss: 0.495612\tvalid_0's auc: 0.768695\n",
      "[5]\tvalid_0's binary_logloss: 0.477462\tvalid_0's auc: 0.768047\n",
      "[6]\tvalid_0's binary_logloss: 0.464938\tvalid_0's auc: 0.766633\n",
      "[7]\tvalid_0's binary_logloss: 0.455599\tvalid_0's auc: 0.769597\n",
      "[8]\tvalid_0's binary_logloss: 0.449157\tvalid_0's auc: 0.769673\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[1]\tvalid_0's binary_logloss: 0.610929\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.556723\tvalid_0's auc: 0.77257\n",
      "[3]\tvalid_0's binary_logloss: 0.521014\tvalid_0's auc: 0.773214\n",
      "[4]\tvalid_0's binary_logloss: 0.495392\tvalid_0's auc: 0.773113\n",
      "[5]\tvalid_0's binary_logloss: 0.477312\tvalid_0's auc: 0.773246\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[7]\tvalid_0's binary_logloss: 0.455974\tvalid_0's auc: 0.773059\n",
      "[8]\tvalid_0's binary_logloss: 0.449487\tvalid_0's auc: 0.771506\n",
      "[9]\tvalid_0's binary_logloss: 0.444703\tvalid_0's auc: 0.771652\n",
      "[10]\tvalid_0's binary_logloss: 0.441448\tvalid_0's auc: 0.771613\n",
      "[11]\tvalid_0's binary_logloss: 0.438751\tvalid_0's auc: 0.77262\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[1]\tvalid_0's binary_logloss: 0.611464\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557735\tvalid_0's auc: 0.76943\n",
      "[3]\tvalid_0's binary_logloss: 0.520989\tvalid_0's auc: 0.769779\n",
      "[4]\tvalid_0's binary_logloss: 0.495424\tvalid_0's auc: 0.771953\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[6]\tvalid_0's binary_logloss: 0.464843\tvalid_0's auc: 0.770981\n",
      "[7]\tvalid_0's binary_logloss: 0.455878\tvalid_0's auc: 0.769906\n",
      "[8]\tvalid_0's binary_logloss: 0.449745\tvalid_0's auc: 0.770067\n",
      "[9]\tvalid_0's binary_logloss: 0.4452\tvalid_0's auc: 0.771402\n",
      "[10]\tvalid_0's binary_logloss: 0.442787\tvalid_0's auc: 0.769621\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[1]\tvalid_0's binary_logloss: 0.611582\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557781\tvalid_0's auc: 0.767552\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[4]\tvalid_0's binary_logloss: 0.495612\tvalid_0's auc: 0.768695\n",
      "[5]\tvalid_0's binary_logloss: 0.477462\tvalid_0's auc: 0.768047\n",
      "[6]\tvalid_0's binary_logloss: 0.464938\tvalid_0's auc: 0.766633\n",
      "[7]\tvalid_0's binary_logloss: 0.455599\tvalid_0's auc: 0.769597\n",
      "[8]\tvalid_0's binary_logloss: 0.449157\tvalid_0's auc: 0.769673\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[1]\tvalid_0's binary_logloss: 0.610929\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.556723\tvalid_0's auc: 0.77257\n",
      "[3]\tvalid_0's binary_logloss: 0.521014\tvalid_0's auc: 0.773214\n",
      "[4]\tvalid_0's binary_logloss: 0.495392\tvalid_0's auc: 0.773113\n",
      "[5]\tvalid_0's binary_logloss: 0.477312\tvalid_0's auc: 0.773246\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[7]\tvalid_0's binary_logloss: 0.455974\tvalid_0's auc: 0.773059\n",
      "[8]\tvalid_0's binary_logloss: 0.449487\tvalid_0's auc: 0.771506\n",
      "[9]\tvalid_0's binary_logloss: 0.444703\tvalid_0's auc: 0.771652\n",
      "[10]\tvalid_0's binary_logloss: 0.441448\tvalid_0's auc: 0.771613\n",
      "[11]\tvalid_0's binary_logloss: 0.438751\tvalid_0's auc: 0.77262\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[1]\tvalid_0's binary_logloss: 0.611464\tvalid_0's auc: 0.764811\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557735\tvalid_0's auc: 0.76943\n",
      "[3]\tvalid_0's binary_logloss: 0.520989\tvalid_0's auc: 0.769779\n",
      "[4]\tvalid_0's binary_logloss: 0.495424\tvalid_0's auc: 0.771953\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n",
      "[6]\tvalid_0's binary_logloss: 0.464843\tvalid_0's auc: 0.770981\n",
      "[7]\tvalid_0's binary_logloss: 0.455878\tvalid_0's auc: 0.769906\n",
      "[8]\tvalid_0's binary_logloss: 0.449745\tvalid_0's auc: 0.770067\n",
      "[9]\tvalid_0's binary_logloss: 0.4452\tvalid_0's auc: 0.771402\n",
      "[10]\tvalid_0's binary_logloss: 0.442787\tvalid_0's auc: 0.769621\n",
      "Early stopping, best iteration is:\n",
      "[5]\tvalid_0's binary_logloss: 0.477466\tvalid_0's auc: 0.77215\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.611582\tvalid_0's auc: 0.7579\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.557781\tvalid_0's auc: 0.767552\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[4]\tvalid_0's binary_logloss: 0.495612\tvalid_0's auc: 0.768695\n",
      "[5]\tvalid_0's binary_logloss: 0.477462\tvalid_0's auc: 0.768047\n",
      "[6]\tvalid_0's binary_logloss: 0.464938\tvalid_0's auc: 0.766633\n",
      "[7]\tvalid_0's binary_logloss: 0.455599\tvalid_0's auc: 0.769597\n",
      "[8]\tvalid_0's binary_logloss: 0.449157\tvalid_0's auc: 0.769673\n",
      "Early stopping, best iteration is:\n",
      "[3]\tvalid_0's binary_logloss: 0.520831\tvalid_0's auc: 0.771232\n",
      "[1]\tvalid_0's binary_logloss: 0.610929\tvalid_0's auc: 0.767714\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.556723\tvalid_0's auc: 0.77257\n",
      "[3]\tvalid_0's binary_logloss: 0.521014\tvalid_0's auc: 0.773214\n",
      "[4]\tvalid_0's binary_logloss: 0.495392\tvalid_0's auc: 0.773113\n",
      "[5]\tvalid_0's binary_logloss: 0.477312\tvalid_0's auc: 0.773246\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[7]\tvalid_0's binary_logloss: 0.455974\tvalid_0's auc: 0.773059\n",
      "[8]\tvalid_0's binary_logloss: 0.449487\tvalid_0's auc: 0.771506\n",
      "[9]\tvalid_0's binary_logloss: 0.444703\tvalid_0's auc: 0.771652\n",
      "[10]\tvalid_0's binary_logloss: 0.441448\tvalid_0's auc: 0.771613\n",
      "[11]\tvalid_0's binary_logloss: 0.438751\tvalid_0's auc: 0.77262\n",
      "Early stopping, best iteration is:\n",
      "[6]\tvalid_0's binary_logloss: 0.464939\tvalid_0's auc: 0.77358\n",
      "[1]\tvalid_0's binary_logloss: 0.619951\tvalid_0's auc: 0.776222\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.569301\tvalid_0's auc: 0.779882\n",
      "[3]\tvalid_0's binary_logloss: 0.533201\tvalid_0's auc: 0.778507\n",
      "[4]\tvalid_0's binary_logloss: 0.506846\tvalid_0's auc: 0.777975\n",
      "[5]\tvalid_0's binary_logloss: 0.487768\tvalid_0's auc: 0.777253\n",
      "[6]\tvalid_0's binary_logloss: 0.473897\tvalid_0's auc: 0.775596\n",
      "[7]\tvalid_0's binary_logloss: 0.462841\tvalid_0's auc: 0.776941\n",
      "Early stopping, best iteration is:\n",
      "[2]\tvalid_0's binary_logloss: 0.569301\tvalid_0's auc: 0.779882\n",
      "Best parameters found by grid search are: {'learning_rate': 0.175, 'n_estimators': 20}\n"
     ]
    }
   ],
   "source": [
    "# Remake our test/train set with our reduced dataset\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=21)\n",
    "\n",
    "reduc_estimator = lgb.LGBMClassifier(learning_rate = 0.125, metric = 'l1', \n",
    "                        n_estimators = 20, num_leaves = 38)\n",
    "\n",
    "# Parameter grid for hyperparameter tuning\n",
    "param_grid = {\n",
    "    'n_estimators': [x for x in range(20, 36, 2)],\n",
    "    'learning_rate': [0.10, 0.125, 0.15, 0.175, 0.2]}\n",
    "\n",
    "gridsearch = GridSearchCV(reduc_estimator, param_grid)\n",
    "\n",
    "gridsearch.fit(X_train, y_train,\n",
    "        eval_set = [(X_test, y_test)],\n",
    "        eval_metric = ['auc', 'binary_logloss'],\n",
    "        early_stopping_rounds = 5)\n",
    "print('Best parameters found by grid search are:', gridsearch.best_params_)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's binary_logloss: 0.649725\tvalid_0's auc: 0.772154\n",
      "Training until validation scores don't improve for 5 rounds.\n",
      "[2]\tvalid_0's binary_logloss: 0.614375\tvalid_0's auc: 0.778549\n",
      "[3]\tvalid_0's binary_logloss: 0.585455\tvalid_0's auc: 0.779084\n",
      "[4]\tvalid_0's binary_logloss: 0.561272\tvalid_0's auc: 0.779688\n",
      "[5]\tvalid_0's binary_logloss: 0.541092\tvalid_0's auc: 0.779204\n",
      "[6]\tvalid_0's binary_logloss: 0.524521\tvalid_0's auc: 0.778438\n",
      "[7]\tvalid_0's binary_logloss: 0.510473\tvalid_0's auc: 0.777338\n",
      "[8]\tvalid_0's binary_logloss: 0.498487\tvalid_0's auc: 0.777082\n",
      "[9]\tvalid_0's binary_logloss: 0.488587\tvalid_0's auc: 0.776332\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's binary_logloss: 0.561272\tvalid_0's auc: 0.779688\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,\n",
       "        learning_rate=0.1, max_depth=-1, metric='l1', min_child_samples=20,\n",
       "        min_child_weight=0.001, min_split_gain=0.0, n_estimators=20,\n",
       "        n_jobs=-1, num_leaves=31, objective=None, random_state=None,\n",
       "        reg_alpha=0.0, reg_lambda=0.0, silent=True, subsample=1.0,\n",
       "        subsample_for_bin=200000, subsample_freq=1)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbm = lgb.LGBMClassifier(learning_rate = 0.1, metric = 'l1', \n",
    "                        n_estimators = 20)\n",
    "gbm.fit(X_train, y_train,\n",
    "        eval_set=[(X_test, y_test)],\n",
    "        eval_metric=['auc', 'binary_logloss'],\n",
    "early_stopping_rounds=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<li>We still need to compare the <b>classification accuracy </b> versus the <b>null accuracy</b> ( the accuracy that could be achieved by always predicting the most frequent class). We must always compare the two.  </li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy of prediction is: 0.8206583427922814\n",
      "The roc_auc_score of prediction is: 0.6585775279927082\n",
      "The null acccuracy is: 0.7790389708664396\n"
     ]
    }
   ],
   "source": [
    "y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration_)\n",
    "print('The accuracy of prediction is:', accuracy_score(y_test, y_pred))\n",
    "print('The roc_auc_score of prediction is:', roc_auc_score(y_test, y_pred))\n",
    "print('The null acccuracy is:', max(y_test.mean(), 1 - y_test.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_prob = gbm.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.35298437, 0.42795826, 0.38211999, ..., 0.40550914, 0.41998315,\n",
       "       0.35931644])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEXCAYAAACu1P9TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8HNXV8PHfkWRZsiW5SW6Su2W5gQEbm47p7QkOCS8lNBMSpxGekISUl4QQkrwPSZ700AyhJbSQkOBQYkoQBoONjRu2sYxcJVdJtmT1et4/7shar1fSSt7VrnbP9/PZjzRlZ87cmT17987MHVFVjDHGxL6ESAdgjDGmZ1jCN8aYOGEJ3xhj4oQlfGOMiROW8I0xJk5YwjfGmDhhCT+GiEiGiPxDRA6JiIrI2EjHdCxEJF9EHmlvOFp1N05vn13fxfccJyIfiEidiGzv6jo7WO5YL54zOplvvog0hWq9x0JE7haRwh5c31HbLiJzRWS9iDR6x0FQ5dhToibhi8jjXsGoiDSLSLGIPCki2QHmHSYifxCR7SLSICIlIvI3ETkhwLxJIvJ170NRKSIVIrJaRO4UkUE9s3U95ivAqcDpwAigKLLhhNxngG+2DojIGyLyeOTCiQq/AA4Bk4GTw7USEcnxPptz/SY9Bxz1GY0Tgbb9AWAVMB53vBbhPovLeza0wKIm4XvewRXOaOBzwInA874ziMgoYCVwGi7BTQQuAxqBZSJysc+8fYCXgZ8BfwXOBWYAdwKnADeFd3OOJCLJYV5FLrBBVT9S1b2q2tydhYQqzlBvr6oeUNVDoVxmd/XAvgxWLvC2qm5X1ZKeXrmq1qrqvp5ebzRoZ9tzgddVtcg7Xpu9z2LjsawrZMebqkbFC3gceMNv3NcBBTJ8xi0C9vqO85n2ijct1Rv+FtACnNrOOgd1EE8ScBewBagHdgF/8JmuwPV+73kDeNxneDvwU+B+oAxYATwFvBZgfa8Cz/oMXwAsBWq9dT8GDOkg3u1eTK2vfG98OvAQUALU4b4sL/R531hv/uu88qsG/reD9VwNfOgtq8yLe5A3LR/4E/ATYA9Q4lOWdwPbvPdtAL7kt9wxwL+97d3p7ft84BGfeQ4Pe8eL+r3mdhD3+bgKRQ1QAbwNTPCmneRtx36gyttPFwco3yP2ZbBxtxPPOcA6rzzWecNHHFPAMG87S4BK73g4y2+/+b7u9qb9DPjY29Yi4EFggM9y5wNNfvHk+Jahz/LP8DnefV/bAy2rdRj3K3OVF8MKYGaA/fGRz/af7b/93diPdwOFPvOOA14AdnvzfwTc4Le8M7xyrfRea4GLfKb/X2ArLgeUAItpyy+Htx2YG6CM5vuXY2f71W9ZlwHvemV067Hk18PLDsVCQhKIX8IHRno7swno740bBDQDP2hnGWd6BXW5N7wGvy+RLsTzBC4B3ABMwP0iuN1nerAJ/5B3IE4CpgIXeduQ7XcANAGXeMPnegfo13E1hpOBt4AlgLQTbxbuJ+YSYDgw2Bv/vBfHRcAU4HdAAzDZ74NdDFyP+yk6rp113Iz7JfVDb1uOB/4byPSm53sH8IPe9ON89u064ELch/BqoBy4xZsuuOSwApgDnAC87pVdewl/gLetz3nbOxxI7iBJNAO/xf3Cmwzc4lMGc3G/9qZ6++mnXhlN6mRfBhV3gHhG4r5YH/OWc4FXPoePKSAV2Aj8HZiF+yV7Jy7xTAESvW0uAu71/k/z3vsD3GdhLHAesAl4wmf98+l6wj/RG/6Mt66sQMvyhlu8fXOmV9av4SpOSd482bjj+xFv+8/DVSI6TPhB7Me7OTLhHwd8DXecTsB9npqAc7zpicAB4Ne4z1kucAVwpjf9M96+/BSu1eEE4BsETvjJXrmot87h3j70L8cO96vP8ajefrsc95nJCUmeDWXSPqZAXFJowtWwamj7lvxfn3lme+OuaGcZg73pd3jDNcDvuxHLRG85V3YwT7AJ/02/eRJwNfbv+oz7Jq5GnOgN5wP3+r1vtLfOEzopwzcCbMelfvOtAh71+2D/MIhy2Qn8sYPp+cBmIMFn3DhcApjsN+9dwBrv//O9GHwTbBau1hww4Qcq7w7iegd4qYvHwFrgzk72ZVBxB1j2T4EdeAnQG/dfHJnw5+O+hJP83vsf4Ld+cQWsAPnMcwUuoST4LLurCf+I6T7vO2JZ3rACJ/mMO8Ubl+cN/8yLO9FnnovpPOF3uB/xS/jtzPMi8LD3/6BA2+Qz7+3e8dynnemByvGIbQhQjp3uV9oS/g0dbUt3XklEl+W4mlYKcBWu5vNDn+nSyfvVb1gCjAvGSd7f17rxXn8f+A6oaouIPIX75fBzb/QNwFPa1uZ+MnCKiNwaYHm5uF8uwZjq/V3iN34J7uRuu3H6E5GhwCg6L5MPVbXFZ3gWbj+sFDli9yXhamutcZaq6ubWiapaIiIFnawrWDOB77U3UUSygB/jflkN92JLwTXX+PIvo+7GPRX4QFV9r/B412+ek71Yyv3KrS/uC6VdIvIZXE10IpCBq2S01kB3dxJbKCjuC7PVLu/vMKAAt/0r9MhzTO8HsdwO96M/EemHq1h8CnduMBlXfm8BqOpB72qqxSLyH1yLwj9UtXX//RW4DdghIq8BbwL/VNXKYGMIoCv7tcPPZHdEW8KvVdXWy6rWi8gk4D7g8964T3C1xenAPwK8f7r3t8Dn77Qwxaoc/QXUJ8B81QHGPQHcISIzcTWvEzjyBHIC7svgzwHeu7froR4l0BdhoDgD6ewL1H85rRcGnIb7xRVoWd39Yu6Kjpb/OO4X1Hdw5xlqgWdxCcKX/7Z1N+5A7/MfTsC1w18R4P3+5di2YJE5uGa8/wHuAA7iathP0LY9LQHeGujY7a4Wv2Teum0JAca1N9yerpT3L4F5uHN5m3D771e45kC3MNUvisjvcM2NFwA/EZFbVfUhVd0lIpNx51fOxVU+fy4ic1S1u1fAdWW/BvuZ7NLKo9ndwE0iMgvcVRq4k2tfE5GMAPP/X2Afrh0V4C/AuSLiX5sFoIPLMld5fy/sILb9uLbY1mX1pa1G3SFV3eCt40bvtUZV1/nMshKYpqqFAV5VwazDs8H7e5bf+DN9pgVFVffjfope1JX34dpmAUYH2JYtPnFmiUhu65tEJBPXVt6RBlw7bDAxdBT3WcD9qrpIVT/CNa+ND2K53Y17AzBHRHxj979Oe6UXw6EA5dZRLf0M3K+OH6jqcu/XR47fPPuBRBEZ5jPuJDrW4P0Nprw7sxE42W/7A35G/XS2H/2dhfvl/JyqrsWdfD1q36jqelX9tapegrvoYIHPtHpV/beqfgd3TqAf8OkuxOCvu/s1JKI64avqJuAlXG2l1ddwTQH/EZGLRWSUiJwsIk/jvonnq2rrT6Pf4X6GLRaRb4vILBEZ473vn7hkG2i9hbirae4XketFZIK3jv/2me0N4MsicqqITMfVErty6dQTwLW4q2Oe9Jt2FzBPRH4jIid4679YRP4kIqnBrsBLqM9723GRiEz2ajPTcbWfrvox8CUR+aGITBGRaSJyq5fk2ouhEHgUeFhEbhCRiSIyQ0Q+LyLf9WZ7E9cE8BcRme3dT/EU7pxOR7YBM73yyfQuww3kJ8AlIvJbETleRPK8m2byvOkFwHXibmI6AXiG4BJbd+N+ANfWv9Arx/Nw7dq+nvK272URudC7gWeOiHxfRDpKOAW4L6FbRGS8iNwIfNVvng9wJ9fvFZFc71LmuzqJuRR3fu1CERl+jPew3I9r3nnA2/5zaNv+jmrwne1HfwW4z9FsEZkKLOTIStpEEfm5iJzh5YVTcZWhjd70W0Tki97xOgb3WU1vnd5N3d2voRHqkwLdfRHgskxv/Om4g+A8n3EjcE09O3A1j1LcWe8TA7w/CXclyUrcT6RDwGrcr4GBHcTTB3eAbffWUcyRJ8uGA//ylleEuycg0Enb9q4oyvSW2wgMCzD9TG95lV7cH+OuTkjqIOajyhDXhtt6WWY97V+WeUZ7y/Vb3nW4JFePuzzx5dZypJ3LEXHJ8zu4n9Wt++tt4P/4xfEa7hK0Ym+fHbG8AMPjcecjquj8ssyLcO3EtbjL+d4CxnvTjgPe86ZtxyXIoPZlMHG3E895uMsE64H1uCYD/xN+Q3BfDru8ctuFa8o8saO4cMftPu+4eQVXsVBgrM88l3nHVC3ussCLfMsw0HGBqyBtwx2z271x8wlwWaZfPEed8MWd8F7vbf862k7afraTcutoP97NkVfpjMJdRlmN+9X2Y1wNPt8nj7zg7bd63PmNh/EuYcVdpfMerlmsxov3lk62tcOTtsHsV9pO2obkyhzfl3grMMaYiBGRs3CVgOPVNauZMLCEb4zpcSLyFdwvxd24c1+/AQ6q6ikRDSzGddqGLyKPish+EVnfznQRkd+LSKGIrBORzk7+GGPMGNyVUAW45o13cM1MJow6reF7P7WqgCdVdXqA6Zfi7mC7FHe34e9UdU4YYjXGGHMMOq3hq+oS3O3H7ZmH+zJQVV0GDBSREaEK0BhjTGiE4sarbI7shrfYG7fHf0YRWYB3jWtKSsrM0aNHh2D1vV9LSwsJCVF9hWyPsbJoY2XRJtbKolmhRaGpBZpVaWlx45oVmr1xrcP+GvYWlqpqVnfWG4qEH6i7g4DtRKq6EHctLHl5eVpQEKo753u3/Px85s6dG+kwooKVRRsrizbRXBZNzS3sqaijvsndwFzX2EzxwVqKD9ZQdKCG4oO1FB2s4WBNIy0tSnltI80tLkUm0NbMkpyYQFZ6XzLTkr2/fY/6m5mWzISh6Tu6G2soEn4x7nrXVjn0TH8dxhjTI1SVksp6tpZWs817bS2pZltpFTsP1NAYqCoOpPdNImdwP8YM6c/MMcmICANT+wRM6BkpSfj1rxNyoUj4i4BbReRZ3EnbClU9qjnHGGOiXUVto5fQq9hWUs3W0mq2l1WzraSa6oa27oGSkxIYN6Q/uUPTuXDacMYO6UdqskunyYlC9sB+jBqcyoDUPmFP4l3RacIXkWdwd35likgx8CO8jpZU9UHcnXyXAoW4u9FuDlewxhgTKrvKa8kv2M+aneWHa+1l1Q2HpycI5Azqx7jM/swaM5jxWf0Zl+leIwekkpAQPYk8WJ0mfFW9tpPpiuvfxhhjokpDUwu7y2vZeaCGooM11De2sPdQHfkF+9m8z/VDmJmWzPisNC6YOuxwQh+f1Z9Rg/vRNykUfcVFj2jrHtkYY4KmqhysaWTngRqX1A/UsLOs5vDwnopaWvya1/skCrPHDeaqWaOYm5fFhKy0qGp2CSdL+MaYqNbSomyvaOZHL64/IjGXVtXzzielVNQe+XzwrPS+jB7cj9njBjNqcD9Ge6+cQan0T06ib58EUvrEVs09WJbwjTERV1HTyPayaooO1lB0wLuk0bu0sfhgLQ1NLcAO+iQK/byTo6l9Erlg6jCmjshwSX2IS+qt083RrGSMMWFX39RMWVUDJZX1lFbVH/5bWtVA8cFa3irYf/jadICB/fowalA/8oalc/6UYdSWFvPps2cyc8zgCG5F72cJ3xhzTPZU1LJpTyUllfWU+CRz36Tu3+zSKj0liay0vlxxYjYXTh3GKK/pJT3lyGfZ5Ofvs2QfApbwjTFBU1V2lbsEv3xbGW8VlFC4/8inbqb1TTp8t+ikYemcPvHou0Vb/4/XtvRIsYRvjAmoqr6Jgr2VbNp7iE172v5W1rsnOCYInJmbxZQRGcwcPZDzpgwjM60vqcmWxKOVJXxj4lxzi7KjrJpNeyvZtOcQH3tJvuhA7eF50lOSmDI8gytOymby8Awmj0hn0rB00vpaCulNbG8ZE4deXreHJZtL2LT3EAX7KqlrdB1/JQiMz0pjRs5Arjl5NJOHpzN5RAYjB6TEzbXqscwSvjFxZtHa3dz2zGpS+iQwc8wgrpszhsnD05kyIoOJQ9OsXT2GWcI3JsbVNTazpqicZVvLWLa1jA93HGTWmEE8s+AU+iTGTh/zpnOW8I2JMf4JftXOchqaWhCBaSMzmH/aWL509gRL9nHIEr4xvVxdYzOrd7Yl+NVFLsEnCEwbOYCbTh3DKeOHMGvsYAak9ul8gSZmWcI3ppexBG+6yxK+MVGquUXZW1HH8m1lPL+ymE17DwFwqLaJhmaX4KdnW4I3wbOEb0wUKdxfyXf+to6dB2ooq2pAX3sTgOyBqZw/ZRiJCUJ6Sh9mjxvErLGDyUixBG+CZwnfmCixrricT9+3lBaFq2blUH9wH7OPz2NiVhonjx3cK5+wZKKLJXxjIqy8poGCvZXc/PgKhmWk8PPPHs9Zk7LIz89n7pwxkQ7PxBBL+Mb0sOYW5UB1A2uLynm3sJTnVxYdfkD2QzfM5MzcrAhHaGKVJXxjetiX//Ihr2/cB0BKnwROHT+Ey44fyaRhaRyfMzDC0ZlYZgnfmB6QX7CfNz/eD8DK7QcAePqLc5g5ZlDMPSjbRC9L+MYco4amFnaUVbO1tJptpdVsK6lmW1k1B6obDs/T2mf84P7JiAh3XJTHaRMyIxWyiVOW8I3phh1l1fz05Y9doi+ppsnn8XyZacmMy+zPpGFpCO7Kmrzh6Vw3ezSnTbQkbyLHEr4xXVBd38Qj72xj4ZItVDc0c2ZuJhdMHUbu0HTGZ/VnbGZ/uzbeRC1L+MYEobG5hRdWFfO/r22mpLKei6YN49sX5pE7LD3SoRkTNEv4xnSgrrGZ5z8s5sH8Lewqr+XE0QN56IaZnDR6UKRDM6bLLOEbE0BtQzNPf7CThUu2sO9QPSeOHshPPz2duXlZ9uQn02tZwjfGs7u8lqWFpby3pYz8gv0crGnklPGD+c1VJ3DqhCGW6E2vZwnfxK3ymgaWbS3j3cJS3issY2tpNeCusjkzN4sbTh3DyWMHRzhKY0LHEr6JG7UNzazccYClhWUsLSxl/e4KVKF/ciJzxg/hc3NGc0ZuJnnD0q02b2KSJXwTs5qaW1i3q4L3Ckt5t7CUVTvKaWhuoU+icOLoQXzjvEmcPnEIM0YNtMf9mbhgCd/EDFWlcH8V7xaWsrSwjOVby6isbwJg6ogM5p8+ltMmDGH2uMH0S7ZD38QfO+pNr7ar9USrd7J1f2U9AGOG9OO/ZozkjImZnDphCIP7J0c4UmMizxK+6VUq6xp5b0sZ73xSwtLCMrb5nGg9bUImp08cwmkTMhk1uF+EIzUm+ljCN1GtpUX5aFcFSzaXsOSTElbtLKe5RQ+faL3+lDGcPnGInWg1JghBJXwRuRj4HZAIPKKq9/pNHw08AQz05vmeqr4S4lhNnGhuUXYdrGVraRUP5G9h+TbXnfD07Ay+dNZ4zpqUxUmjB5GcZCdajemKThO+iCQC9wEXAMXAChFZpKobfWb7AfBXVX1ARKYCrwBjwxCviXEf7mviq3cvpsZ7AhTAHRflcfXJo8hM6xvByIzp/YKp4c8GClV1K4CIPAvMA3wTvgIZ3v8DgN2hDNLEh32H6nh0fT1jMzO48dQxjMvsT/agVHIGWXu8MaEQTMLPBop8houBOX7z3A28JiJfB/oD5wdakIgsABYAZGW5hzQbqKqqiuuyUFU2lLXw8tYGGpqVGyc2MrxmK7U7oXAnFEY6wAiJ9+PCl5VFaAST8AOdCVO/4WuBx1X1VyJyKvBnEZmuqi1HvEl1IbAQIC8vT+fOnduNkGNPfn4+8VoWqsqP/7WRx1duB+DGqX255rJzIxtUlIjn48KflUVoBJPwi4FRPsM5HN1kcwtwMYCqvi8iKUAmsD8UQZrY1NKi3LVoPX9ZtpPPnz6OW84cxydrlkc6LGNiVjCXOawAckVknIgkA9cAi/zm2QmcByAiU4AUoCSUgZrY0tKi3PnPj/jLsp186ezx/PC/ppA9MDXSYRkT0zqt4atqk4jcCizGXXL5qKpuEJF7gJWqugj4FvCwiNyOa+6Zr6r+zT4mTu0ur2XngRp2Hqhh9c5yVu88SOH+KppalFvPmci3Lpxk19Ab0wOCug7fu6b+Fb9xd/n8vxE4PbShmVjw4Y4DfO7h5dQ3udM5GSlJnDB6EGfnZTF1RAaXzxhpyd6YHmJ32pqwKdxfyecfX8mIASncM286IwemMj6zPwkJluCNiQRL+CYs9lbUcdOjK+iTmMCTn5/D6CF2Lb0xkWb3ppuQq6htZP5jH1BR28jjN59syd6YKGEJ34RUXWMzC55cyZaSKh68fibTswdEOiRjjMeadEzINLco3/zrGpZvO8DvrjmBM3IzIx2SMcaH1fBNyPzkpY288tFefnDZFOadkB3pcIwxfqyGb45ZfVMzv359M4+/t53Pnz6OL5w5PtIhGWMCsIRvjknRgRpufWY1a4vKOXX8EL5/6eRIh2SMaYclfNNtHxVXcPPjKzhU18gPLpvC508fZ9fYGxPFLOGbLttWWs2f39/Bsyt2MqhfMq/cdgYTh6ZHOixjTCcs4Zug1TY086vXCnji/e2ICOfmDeWeedMYmpES6dCMMUGwhG861ZroH3l3GwBXzxrFty6axNB0S/TG9CaW8M0RmppbOFDdwO6KOjbsrmDj7kO8vbmE4oO1XD5jJOdMzuKKE3MiHaYxphss4ZvDVu88yDeeW8OOsprD49JTkpg2MoMfXDaVi6cPj2B0xphjZQnfUNfYzCPvbOU3b3zC8IwUfvSpqYwYkMK0kQPIGZRq3RcbEyMs4ce5Nz/ex23PrKa6oZlPzRjJTz89nQGpfSIdljEmDCzhx6lDdY08vXwn9766if7JiTz1hTmcNmGI1eaNiWGW8ONMc4vypT9/yBsf7wPg9IlD+OYFk5g5ZnCEIzPGhJsl/DjS2NzCTY9+wHtbyhg5IIWFN86y7ouNiSOW8OOAqvLv9Xv5xeICtpVWM/+0sfzoU1Ot+caYOGMJP0bVNzXzyDvbKNhbybbSaj7aVUHu0DT+dNMszp081JK9MXHIEn6MOVjdwB/fKuSFVcUcrGkkM60vWel9+flnj+OzJ+WQlGiPQDAmXlnCjyGLN+zlzn+sp7ymgYumD+fqWaM4MzfTavPGGMASfkw4WN3A3f/awItrdjN1RAZPfn42U0dmRDosY0yUsYTfyy1csoWH3t5KRW0jt58/ia+eM4E+1mxjjAnAEn4v9NK63Tz8zjYq6xrZWlJNgsC/vn4G00baJZbGmPZZwu9Fquqb+NGLG/j7qmLyhqUzPjONnEH9+O/zci3ZG2M6ZQk/ypVV1bPgzx9SsLeShuYWmppbuO3cidx2Xq5dcWOM6RJL+FHsUF0j1z2ynG2l1Vw7ezRJCcIlxw23bhCMMd1iCT9K7amo5eevbmLT3kr+cO2JfGrGyEiHZIzp5SzhR6GtJVV84cmVbC2pZtKwNC6aZg8eMcYcO0v4UebDHQf57APvAfCdi/P4whnjSU6ytnpjzLGzhB9FnvlgJ3e9uJ7MtGTu+9xJzBk/JNIhGWNiiCX8KPLwkq00Nit/+/JpjM3sH+lwjDExJqi2AhG5WEQKRKRQRL7XzjxXichGEdkgIk+HNszY1tCsPPT2FraWVnPPvGmW7I0xYdFpDV9EEoH7gAuAYmCFiCxS1Y0+8+QC3wdOV9WDIjI0XAHHkrrGZh5duo1H3qnlQN0mzszN5NMnZkc6LGNMjAqmSWc2UKiqWwFE5FlgHrDRZ54vAvep6kEAVd0f6kBjzV+W7eAH/1wPQE6a8IcvzOH0iZkRjsoYE8uCSfjZQJHPcDEwx2+eSQAishRIBO5W1X/7L0hEFgALALKyssjPz+9GyL3fmv1N/HZVPQAXj03i0uxGGovXk18c4cCiQFVVVdweF/6sLNpYWYRGMAk/UGfqGmA5ucBcIAd4R0Smq2r5EW9SXQgsBMjLy9O5c+d2Nd5e70B1A9/+zRLGZfbnL1+YQ/bAVPLz84nHsgjEyqKNlUUbK4vQCOakbTEwymc4B9gdYJ4XVbVRVbcBBbgvAOPnrhfXU1HbwP3XnUT2wNRIh2OMiSPBJPwVQK6IjBORZOAaYJHfPP8EzgEQkUxcE8/WUAYaCzbvq+SldXv4ytkTmDLCHlBijOlZnTbpqGqTiNwKLMa1zz+qqhtE5B5gpaou8qZdKCIbgWbgDlUtC2fgvcX+Q3W88tEe3t9axuIN+0hOSuCm08ZGOixjTBwK6sYrVX0FeMVv3F0+/yvwTe9lPKt3HuSK+987YtwPL5vCkLS+EYrIGBPP7E7bMHlj4z5ufWYVOYNSue3cXC6YOoz+fZOsXxxjTMRYwg+D51cW8d2/r2N69gAenX8ymVajN8ZEAUv4IVZe08DdizYweXgGz3zxFPr3tSI2xkQHa18IoeYW5Xt//4i6phZ+ceXxluyNMVHFMlIIbCmp4rGl21i8YR8llfVcO3sU07PtoeLGmOhiCf8YVdQ0ct3Dy9l7qI5LjxvOJdNHcMl0e0KVMSb6WMI/Rj9atJ7Sqnpe/NrpzBg1MNLhGGNMuyzhd1Fzi7J5XyUrtx/gvS1lvLp+L7efP8mSvTEm6lnC74KKmkZufHQ5a4srDo+bOiKDr54zIYJRGWNMcCzhB6mippHr/7Scgr2VfP+SyVx63AhGDkwlQUAkUIeixhgTXSzhB8E32T94w0mcO3lYpEMyxpgus+vwO1FR08gNj7pk/8D1luyNMb2XJfwOtCb7j/cc4oHrT+K8KZbsjTG9lyX8dtQ0NHHjYx/w8Z5DPHj9TEv2xphez9rwA1i18yC3PbOa3eW1PGDJ3hgTIyzh+3h94z5eWFXM+1vLaGlRfnP1CVw0ze6aNcbEBkv4ngPVDdz69Cr6JCYwdWQGv7zyeMYM6R/psIwxJmQs4Xue+WAn9U0tvPT1M8gdlh7pcIwxJuTspC3wj9XF/HJxAadPHGLJ3hgTs+K+hv/g21u499VNjByQwk2njo10OMYYEzZxm/BVlfe3lHHvq5tIT0nine+eS2KCdZFgjIldcZfwW1qUNzft5763CllTVE5mWjJ3XJRnyd4YE/PiJuE3tyhPvLed/3n1YxqblVGDU/nZFdP57Ek5pPRJjHR4xhgTdnGT8O9/q5Bfvb4ZgB9fPo3r5owmKdHOWRtj4kfcJPz/FOwHYPE3ziJvuF2JY4yJP3FRxf14zyFW7yznuxehGt1sAAATHUlEQVRPtmRvjIlbcZHwH1u6jdQ+iVw7e1SkQzHGmIiJ+YT/XmEpf11ZzGdnZjOwX3KkwzHGmIiJ6YT/wbYDzH9sBSIw/7RxkQ7HGGMiKmYT/t6KOq566H1yBqXy9rfPYeLQtEiHZIwxERWzCf/vq4oB+PG8aYwe0i/C0RhjTOTFbMJ/df0eZuQM4MzcrEiHYowxUSEmE/6WkirW7zrEp2aMjHQoxhgTNWIu4W8vreaK+5YCWMI3xhgfMZfw71q0gUN1Tdx+/iSGZaREOhxjjIkaQSV8EblYRApEpFBEvtfBfFeKiIrIrNCFGLxVOw+yZHMJd1yUx3+fnxuJEIwxJmp1mvBFJBG4D7gEmApcKyJTA8yXDtwGLA91kMG699VNJCYIn5s9OlIhGGNM1Aqmhj8bKFTVraraADwLzAsw30+AXwB1IYwvaPsr61i14yCXzxjJoP52R60xxvgLprfMbKDIZ7gYmOM7g4icCIxS1ZdE5NvtLUhEFgALALKyssjPz+9ywO156uN6WlQ5Je1ASJfbE6qqqnpdzOFiZdHGyqKNlUVoBJPwAz0KSg9PFEkAfgPM72xBqroQWAiQl5enc+fODSrIzpRW1fPG4jf4PzNzuPqyGSFZZk/Kz88nVGXR21lZtLGyaGNlERrBNOkUA77dTOYAu32G04HpQL6IbAdOARb15Inb+94qRBVuOGVsT63SGGN6nWAS/gogV0TGiUgycA2wqHWiqlaoaqaqjlXVscAy4HJVXRmWiP2oKm9t2s/YIf2Ynp3RE6s0xpheqdOEr6pNwK3AYuBj4K+qukFE7hGRy8MdYGfe+Hg/28tq+MrcCYjYg8iNMaY9QT3iUFVfAV7xG3dXO/POPfawgrNy+wG++ORKRg/uxxUn5vTUao0xplfqtc+0/fXrm/n9m5+QmZbMbeflkpwUczcNG2NMSPXKhF9d38Rj725j2sgMnv/yqfRL7pWbYYwxPapXVotfWL2Lyvom7pk33ZK9McYEqdclfFXlyfe2Mz07g5NGD4x0OMYY02v0uoT//tYyPtlfxU2njrWrcowxpgt6XcJ/8r0dDOrXx/q6N8aYLupVCX9XeS2vbdzL1SePJqVPYqTDMcaYXqVXJfynlu0A4Lo51v2xMcZ0Va9J+HWNzTy7oojzpgxj1OB+kQ7HGGN6nV6T8F9et4cD1Q3MP21spEMxxpheqdck/Cff386ErP6cNmFIpEMxxpheqVck/DVF5awtruCm0+xSTGOM6a5ekfCXFpYCMO+E7AhHYowxvVevSPhPL98JQL9kuxTTGGO6q1ck/AGpfUhOSqBPYq8I1xhjolKvyKAVtY1cNG14pMMwxpheLeoT/oHqBnaV1zJ9pD2+0BhjjkXUJ/yPdlUAcFzOgAhHYowxvVv0J/zicgCmZ1vCN8aYYxH1CX9tcQXjM/uTkdIn0qEYY0yvFvUJf8OuCqvdG2NMCER1wq9vamZ3RR2ZaX0jHYoxxvR6UZ3w1xW7E7bZg1IjHIkxxvR+UZ3w1xa5E7aX29OtjDHmmEV1wl9TVE72wFSy0q1JxxhjjlVUJ/y1xeXMGGUnbI0xJhSiNuGXVdVTdKCWGTkDIx2KMcbEhKhN+K0nbGeMsoRvjDGhELUJf01ROQkCx9k1+MYYExJRm/DXFpeTOzSd/n2TIh2KMcbEhKhM+KrKuuIKO2FrjDEhFJUJv/hgLQeqG6z93hhjQigqE/4a74Yru0LHGGNCJyoT/tqicvomJZA3PD3SoRhjTMwIKuGLyMUiUiAihSLyvQDTvykiG0VknYi8KSJjjiWotcXlTBuZYc+wNcaYEOo0o4pIInAfcAkwFbhWRKb6zbYamKWqxwN/A37R3YCamlv4aFeFtd8bY0yIBVOFng0UqupWVW0AngXm+c6gqm+pao03uAzI6W5Am/dVUdfYwgmW8I0xJqSCucg9GyjyGS4G5nQw/y3Aq4EmiMgCYAFAVlYW+fn5R82TX9QIQN2uAvLLPwkivN6vqqoqYFnEIyuLNlYWbawsQiOYhC8BxmnAGUWuB2YBZwearqoLgYUAeXl5Onfu3KPm+fff1zEgdS9XXXoOIoFWHXvy8/MJVBbxyMqijZVFGyuL0Agm4RcDo3yGc4Dd/jOJyPnAncDZqlrf3YDWFrv2+3hJ9sYY01OCacNfAeSKyDgRSQauARb5ziAiJwIPAZer6v7uBlPT0MTmfZWckGN32BpjTKh1mvBVtQm4FVgMfAz8VVU3iMg9InK5N9svgTTgeRFZIyKL2llchzbsPkRzi9oVOsYYEwZB9Uymqq8Ar/iNu8vn//NDEUzrIw2PtztsjTEm5KLqziZ7pKExxoRPVCV8e6ShMcaET9QkfHukoTHGhFfUJHx7pKExxoRX1CR8e6ShMcaEV9Qk/HX2SENjjAmrqEj4qurdYWu1e2OMCZeoSPj2SENjjAm/qEj49khDY4wJv6hI+PZIQ2OMCb+oSPib9laSNzzdHmlojDFhFBUZtmBfJZOGWe3eGGPCKeIJ/0B1AyWV9eRZwjfGmLCKeMLfvK8SgEnWfm+MMWEV8YT/iZfwrYZvjDHhFfGEX7CvkoyUJIZlWJfIxhgTThFP+Jv3VpE3PN2eYWuMMWEW0YSvqhTsqyTXmnOMMSbsIprw91fWU1HbaO33xhjTAyKa8Av2elfoWMI3xpiwi2jCP3xJ5rC0SIZhjDFxIeIJPzOtL0PS7AodY4wJt8g26eyrIm+41e6NMaYnRDThf7Kvktyh1n5vjDE9IWLPE2xqgfqGZusS2RhjekjEavgNLQrYFTrGGNNTIpbwG5vdX7tCxxhjekYEa/iQPTCV9JQ+kQrBGGPiSgRr+Equ1e6NMabHRCzhtygM6pccqdUbY0zciXhvmcYYY3pG5Gr4QGpyYqRWb4wxcSeiTTrZA1MjtXpjjIk7EW3SyRlkCd8YY3pKRBO+1fCNMabnBJXwReRiESkQkUIR+V6A6X1F5Dlv+nIRGRvMcrOthm+MMT2m04QvIonAfcAlwFTgWhGZ6jfbLcBBVZ0I/Ab4eTArH5qe0rVojTHGdFswNfzZQKGqblXVBuBZYJ7fPPOAJ7z//wacJ508lTxJIDHBHlxujDE9JZjeMrOBIp/hYmBOe/OoapOIVABDgFLfmURkAbDAG6wXkfXdCToGZeJXVnHMyqKNlUUbK4s2ed19YzAJP1A1XLsxD6q6EFgIICIrVXVWEOuPeVYWbaws2lhZtLGyaCMiK7v73mCadIqBUT7DOcDu9uYRkSRgAHCgu0EZY4wJvWAS/gogV0TGiUgycA2wyG+eRcBN3v9XAv9R1aNq+MYYYyKn0yYdr03+VmAxkAg8qqobROQeYKWqLgL+BPxZRApxNftrglj3wmOIO9ZYWbSxsmhjZdHGyqJNt8tCrCJujDHxwXrLNMaYOGEJ3xhj4kTYE364umXojYIoi2+KyEYRWScib4rImEjE2RM6Kwuf+a4UERWRmL0kL5iyEJGrvGNjg4g83dMx9pQgPiOjReQtEVntfU4ujUSc4SYij4rI/vbuVRLn9145rRORk4JasKqG7YU7ybsFGA8kA2uBqX7zfBV40Pv/GuC5cMYUqVeQZXEO0M/7/yvxXBbefOnAEmAZMCvScUfwuMgFVgODvOGhkY47gmWxEPiK9/9UYHuk4w5TWZwFnASsb2f6pcCruHugTgGWB7PccNfww9ItQy/VaVmo6luqWuMNLsPd8xCLgjkuAH4C/AKo68ngelgwZfFF4D5VPQigqvt7OMaeEkxZKJDh/T+Ao+8JigmquoSO72WaBzypzjJgoIiM6Gy54U74gbplyG5vHlVtAlq7ZYg1wZSFr1tw3+CxqNOyEJETgVGq+lJPBhYBwRwXk4BJIrJURJaJyMU9Fl3PCqYs7gauF5Fi4BXg6z0TWtTpaj4Bguta4ViErFuGGBD0dorI9cAs4OywRhQ5HZaFiCTgel2d31MBRVAwx0USrllnLu5X3zsiMl1Vy8McW08LpiyuBR5X1V+JyKm4+3+mq2pL+MOLKt3Km+Gu4Vu3DG2CKQtE5HzgTuByVa3vodh6WmdlkQ5MB/JFZDuujXJRjJ64DfYz8qKqNqrqNqAA9wUQa4Ipi1uAvwKo6vtACq5jtXgTVD7xF+6Eb90ytOm0LLxmjIdwyT5W22mhk7JQ1QpVzVTVsao6Fnc+43JV7XanUVEsmM/IP3En9BGRTFwTz9YejbJnBFMWO4HzAERkCi7hl/RolNFhEXCjd7XOKUCFqu7p7E1hbdLR8HXL0OsEWRa/BNKA573z1jtV9fKIBR0mQZZFXAiyLBYDF4rIRqAZuENVyyIXdXgEWRbfAh4WkdtxTRjzY7GCKCLP4JrwMr3zFT8C+gCo6oO48xeXAoVADXBzUMuNwbIyxhgTgN1pa4wxccISvjHGxAlL+MYYEycs4RtjTJywhG+MMXHCEn4vJyLNIrLG5zW2g3nHttf7XhfXme/1aLjWu90/rxvL+LKI3Oj9P19ERvpMe0REpoY4zhUickIQ7/mGiPTrxrp+KyJnef/f6vViqN51811dVp4X+xoR+VhEQvq0JxG5vLUnShHJEtdL7WoROVNEXhGRgR28t9391sF73hCRQaHbAtNtke4Vzl7H3KteVRfmHUs7ve91cZ35eL1XAguARaFaXojLxjfOm4HXg3jPdiCzi+sZDCzzGT7RK+suL8t7/2Jgns/wcWE8fq4BngjnfsPdWHlnuLbBXsG/rIYfg7ya/Dsissp7nRZgnmki8oFXi1wnIrne+Ot9xj8kIomdrG4JMNF773leTfEjcf159/XG3ytt/fz/rzfubhH5tohcies36Clvnale7XaWiHxFRH7hE/N8EflDN+N8H5/OpUTkARFZKa5/+R97424DRgJvichb3rgLReR9rxyfF5G0AMu+Evh364CqrlbV7Z3E05ERuFvnW5f3kRfLfBF5UUT+7f1y+ZHP9gQsD3H9y6/yfuW86bOcP3q/eH4BXOpT9ttbf5WIyI3ePlsrIn/2xrW33y4TkX/4xHOBiLzgDS7C9YFjIi3S3zj2OrYX7s7LNd7rH964fkCK938u7i5F8KnhA38ArvP+TwZSgSnAv4A+3vj7gRsDrDOftprzHcBzuFvci4BJ3vgngW/gar8FtN3kN9D7ezfwbf/l+Q4DWbjuclvHvwqc0c04vwH8P59pg72/id58x3vD2/Fq5bg+WpYA/b3h7wJ3BVjPE8CnAow/vKwu7tObcb3Gvgrc7lNm84E9uN5kU4H1XjkFLA+v/IqAcX7bPB/4o///vjED07z9lun33oD7DdeZ1yYgyxt+2rdMgE+AIZH+vMT7K9y9ZZrwq1VV/7bpPkBrDa4Z1/eKv/eBO0UkB3hBVT8RkfOAmcAKcV07pALt9enzlIjU4hLE14E8YJuqbvamPwF8Dfgjrj/7R0TkZSDo7o5VtUREtorrK+QTbx1LveV2Jc7+uMTu+1Sgq0RkAa57kRG4h2ms83vvKd74pd56knHl5m8EIezPRVUfE5HFwMW4fs+/JCIzvMmvq9etgleDPgNoInB5nAIsUdfhGqralU4JzwX+pqqlwbxXVdX7FXC9iDwGnIr70mm1H/frKea6hOhNLOHHptuBfcAM3In5ox4goqpPi8hy4DJgsYh8AVdLe0JVvx/EOq5Tn87MRCTgMwzU9Y8yG9fh1TXArbhkEqzngKtwtcd/eImlS3Hinpx0L3Af8BkRGQd8GzhZVQ+KyOO4Xyj+BJdgO2uOqG3n/e3ykuKJwG5VPeoxfaq6G3gUeFTcifbprZP8Z6Wd/SYilweYP+gQu/Hex3C/NOqA59U936JVCq6cTARZG35sGgDsUddH+A242u0RRGQ8sFVVf49rYz0eeBO4UkSGevMMluCfq7sJGCsiE73hG4C3vTbvAar6Cq5ZJdCVMpW4LpEDeQH4NK4N+DlvXJfiVNVG4AfAKeJ6WMwAqoEKERkGXNJOLMuA01u3SUT6iUigX0sf453HCJaq3qyqJwRK9l67ex/v/+G4Jpxd3uQLvO1NxZXLUtovj/eBs70vOERkcBdCfBP3K2hIB+89Yr95X1K7cWX9uM/2CDAc92vQRJAl/Nh0P3CTiCzDNedUB5jnamC9iKwBJuMel7YR92F9TUTWAa/jmis6pap1uLbn50XkI6AFeBCXEF7ylvc27teHv8eBB1tPHPot9yCwERijqh9447ocp6rWAr/CtT+vxT0jdgOuFr3UZ9aFwKsi8paqluDauJ/x1rMMV1b+Xsb1bAi4k7/iejjMAdaJyCMdxRbAhbh9sxZ3xc4dqrrXm/Yu8GfcOZu/q+rK9srDi38B8IK3rOf8V9QeVd0A/Az3pb0W+HWA2R7n6P32FFDkxdRqJu4qpib/BZieZb1lGhMCIvIu8F8axqdQich83EnSW8O1jmMlIn8EVqvqn3zG/Q536e6bkYvMgNXwjQmVbwGjIx1EJInIh7imwb/4TVpvyT46WA3fGGPihNXwjTEmTljCN8aYOGEJ3xhj4oQlfGOMiROW8I0xJk78f/X1Fp6L9w8XAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "\n",
    "plt.plot(fpr, tpr)\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.0])\n",
    "plt.rcParams['font.size'] = 12\n",
    "plt.title('ROC curve for credit card defaulting classifier')\n",
    "plt.xlabel('False Positive Rate (1 - Specificity)')\n",
    "plt.grid(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC with dimensionality reduction: \n",
      "0.7796879885833661\n",
      "AUC without dimensionality reduction: \n",
      "0.7686460477376955\n"
     ]
    }
   ],
   "source": [
    "auc_roc_1 = str(roc_auc_score(y_test, y_pred_prob))\n",
    "print('AUC with dimensionality reduction: \\n' + auc_roc_1)\n",
    "print('AUC without dimensionality reduction: \\n' + auc_roc_0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Conclusion</h2>\n",
    "We were able to reduce the size of our training/test set by 26% by removing 6 features, while only giving up 0.52% in our AUC accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "- AUC is useful as a single number summary of classifier performance\n",
    "- Higher value means that it is a better classifier\n",
    "- If you randomly chose one positive and one negative observation, AUC represents the likelihood that your classifier will assign a higher predicted probability to the positive observation\n",
    "- AUC is useful even when there is high class imbalance (unlike classification accuracy)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a good thing that the prediction accuracy is greater than the null accuracy because it shows us that the model is performing better than by just predicting the most frequent class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Confusion matrix </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1954  105]\n",
      " [ 369  215]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print(metrics.confusion_matrix(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<b>Basic terminology</b>\n",
    "\n",
    "- <b>True Positives (TP)</b>: we correctly predicted that they would default. \n",
    "    - 233\n",
    "     \n",
    "     \n",
    "- <b>True Negatives (TN)</b>: we correctly predicted that they won't default.\n",
    "    - 2252\n",
    "     \n",
    "    \n",
    "- <b>False Positives (FP)</b>: we incorrectly predicted that they did default.\n",
    "    - 108\n",
    "    - Falsely predict positive\n",
    "    - Type I error\n",
    "      \n",
    "       \n",
    "- <b>False Negatives (FN)</b>: we incorrectly predicted that they didn't default. \n",
    "    - 407\n",
    "    - Falsely predict negative\n",
    "    - Type II error\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
